Chapter 0 - Design of Web Applications

By John Lenz. 2016.

Modern dynamic websites are designed around the browser's document object model (DOM). The DOM is a tree of elements such as text, buttons, checkboxes, textboxes, invisible sections of the page, and many other widgets that the browser uses to display the page to the user. The DOM is initially created from the HTML document and then evolves as JavaScript code executes. JavaScript code can manipulate the properties of DOM elements, create or destroy DOM elements, attach event handlers to DOM elements, and communicate over the network, among other things.

The central question is "Where does the transformation between domain data and the DOM live?" Consider Reddit: the domain data stored by Reddit is the subreddits, the stories, the comments, the votes, the user preferences, the currently logged in user (if any), and so on. We can view the task of a web developer as writing a function which maps this data to the eventual DOM that will be displayed in the user's browser. The different designs of web applications focus on the different possibilities for this mapping of domain data to the DOM.

  1. The original design of HTML, CSS, and JavaScript works as follows: the HTML contains the content, the CSS describes how the content is presented, and JavaScript permits dynamic behavior. On each request, the server generates the HTML document based on the domain data. A good example of this design is Wikipedia. When your browser requests the Wikipedia homepage, the Wikipedia server queries a database and then generates the HTML document containing the title, story, subtitles and so on. The server also returns a CSS document which specifies how to display this content. (For example, each subtitle is in a larger font and directly above a horizontal line across the page.)

    In this design, JavaScript is used for simple user-experience improvements and does not have access to the domain data. JavaScript is typically used for tasks such as form validation, dynamic menus, image galleries, and tabs. For example, consider an image gallery where the user can page through a collection of images via next and previous buttons (here is a good example). When generating the HTML document, the server included all the images and captions. JavaScript code then attaches to events on the next and previous button and controls which DOM elements are currently visible. As another example, consider a credit card input form on a store's website. The Luhn algorithm validates credit cards, so JavaScript code can attach to a button's click event, validate the card number, and if it is invalid manipulate the DOM to make the input box's background color red.

    In this first design, all the domain data used to create the DOM lives on the server, and if JavaScript is used at all it is for manipulation of the DOM to improve the user experience.

  2. The second design is very similar, except now the JavaScript code has access to the domain data used to create the DOM. Reddit is an example of this design. On a story with many comments, the Reddit server will not return all the comments in the HTML document. Instead, inside deeply nested threads, the server will create an HTML element with the text "load more comments". (This story, which was on the front page as I write this, is a good example.) JavaScript will attach to the click event on the "load more comments" DOM element and when clicked will send a network request directly to the server. The server responds with the original domain data, not with HTML. In Reddit's case, the server responds with a JSON document containing the comments. The JavaScript code then creates DOM elements for the comments.

    In this second design, the transformation from the domain data to the DOM happens both on the server and the client.

  3. The final option is for the transformation from domain data to the DOM to occur entirely in JavaScript on the client. Gmail, Twitter, and Facebook are big examples of this design. Here, the server returns an HTML document with no content or possibly just a single "Loading" text element. The initial DOM therefore is either blank or just a single "Loading" element (you can see this when you first open Gmail). Once the JavaScript code executes, it first connects to the server to obtain the domain data and then uses that domain data to create the DOM for the entire page. For example, once the JavaScript on Gmail's page starts executing, it connects to Google's servers and receives the list of messages, folders, and so on and generates the entire DOM.

    In this third design, the transformation from the domain data to the DOM happens only on the client.


My recommendation for Haskell is to use either Design 1 or Design 3 (or both for different parts of the website). I have built several successful projects using Design 2 in Haskell, but I now believe that keeping the transformation from domain data to DOM in one place (either the server or browser) has large advantages in coding and maintenance (at least in Haskell).

Design 1 supports the most browsers, from ancient browsers to limited browsers running on tiny hardware to the latest and greatest browsers running on modern hardware. For example, Wikipedia's goal is to be accessible to anyone with an internet connection so firmly stays in Design 1. Design 1 also has a large amount of tooling, libraries, and development resources available. As a disadvantage, Design 1 cannot implement highly dynamic websites where parts of the DOM change as the domain data changes. The main danger is feature creep, where aspects of Design 2 enter into the website over time. Design 2 can work, but only if the domain data and architecture is designed from the start.

Design 3 is by far the cleanest, simplest design for dynamic websites. As Haskellers, we view the transformation from domain data to DOM as conceptually a pure function. When we keep this transformation localized to the browser it can be implemented as an actual pure function. Consider Gmail. The Gmail server does not know if you are currently viewing your inbox, composing a message, or reading an existing message. When a new email arrives, the Gmail server can just notify the browser that a new email arrived. If you are currently looking at your inbox, JavaScript can create a new DOM element for the new email. If you are currently composing a message, the newly received email can be stored in memory and displayed when you return to your inbox. Most importantly, the current user state does not need to be shread with the server. By keeping most of this state private to the client JavaScript code, we can avoid many of the fallacies of distributed computing and distributed state. The only thing shared between the client and server is domain data, which can be designed around the network boundary.

Design 3 requires modern browsers, is more complicated to integrate with search engines, and is still somewhat new so tools and libraries are growing quickly. Design 3 requires most of the new browser features added as part of HTML5 in the past few years; the main problem is Internet Explorer. Generally IE 9 and below do not have enough features and IE 10 is possible to support but quite difficult. So typically Design 3 websites will only support IE 11 and up. The good news is that IE 11 and up are the only browsers receiving security updates and IE 11 is available for Windows 7 and all later versions, so IE 11+ captures the majority of users who have a modern, up-to-date Windows OS.

The Google web crawler executes JavaScript and extracts content from the resulting DOM, so a Design 3 website will be visible in Google. For other web crawlers, less is known. As of 2016, Bing/Yahoo executes some JavaScript but does not execute everything and other web crawlers likely do not execute JavaScript. There are several strategies to make Design 3 websites visible to all crawlers, but they introduce complexity. If the highly dynamic portion of the website is behind a user login and should not be accessed by web crawlers, Design 3 can easily be used for the dynamic portion of the site without caring about search engines, while Design 1 can be used for the public portion of the site.

Finally, the main benefit of Design 2 is progressive enhancement. For example, if you disable JavaScript and then visit Reddit, the site will still mostly work. You can still view stories, comment, and vote on stories and comments. This comes at a cost of maintaining distributed state about the current user interaction. Also, DOM elements are now created in two places; on Reddit, the DOM elements for most comments are created from HTML generated by the server, except when you click the "More comments" element and then DOM elements are created by JavaScript. For the most part CSS is used to make sure the comments all look the same, although it complicates development since any changes or improvements must be made in the server code and the client JavaScript. There are now many JavaScript libraries to help manage the distributed state and synchronized DOM generation which mostly overcomes this limitation. Unfortunately, these JavaScript frameworks do not integrate well with Haskell. Haskell is excellent at pure data transformation and is good at distributed state when both the client and server are both Haskell, but bridging between a JavaScript library on the client and a Haskell server is clunky.

Haskell Tools

Yesod is an excellent library for building websites in Design 1 and 2 which I have used successfully in the past. While I haven't personally used them, Snap and happstack are also great libraries for Design 1 and 2, and many other smaller libraries.

For Design 3, there is not currently a framework or all encompassing library in Haskell. Instead, there are several great pieces that can be combined and this will be the focus of the remainder of this book.