Chapter 6 - Acceptance Testing

By John Lenz. 2016.

Acceptance testing is black-box testing where the tests interact with the full program/web site exactly as a user does. Acceptance testing will run the production build of the server and load the metalsmith static site into a real web browser such as Firefox and Chrome. The acceptance tests then click elements on the page, send keyboard keys, and make sure the DOM is as expected. Importantly, the only interaction the acceptance test has is via the browser. Selenium and specifically hspec-webdriver make writing these kinds of tests quite easy.

What should the tests focus on? The datastorage library and the servant server both have been extensively unit-tested so we can be reasonably confident that this code works. I am also reasonably confidant that the react-flux stores work correctly: the network communication is type-checked by the react-flux-servant library and if the store contains a complex domain data transformation, I usually move the transformation function into the domain library and test it there. The only thing left is the react views: is the correct data being displayed and do all the react components function properly? Do the views interact with native JavaScript components properly?

Project Setup

The test suite lives in an acceptance-test subdirectory and consists of a separate cabal project. The reason for this is that the acceptance-test cabal project does NOT depend on any server code. By making a separate cabal project, it lessens the temptation to cheat and forces the website to be tested exactly as a user will interact with the site. This acceptance-test cabal project is built via stack and depends just on hspec-webdriver, webdriver, and hspec.

One crucial file in the test suite is WebdriverExtended.hs. In this file, I have some helpers for various tasks missing from the base webdriver package. Here is a fragment:

waitFor :: String -> WD a -> WD a
waitFor message w = waitUntil 10 w
    `catch` \e@(FailedCommand ty _) -> if ty == Timeout then error ("Waiting for " ++ message ++ " timed out") else throwM e

waitForDisplayed :: Maybe Element -> Text -> WD ()
waitForDisplayed from css =
    waitFor ("element " ++ unpack css ++ " to be displayed") $ do
        e <- maybe findElem findElemFrom from $ ByCSS css
        isDisplayed e >>= expect

waitForRemoved :: Maybe Element -> Text -> WD ()
waitForRemoved from css = waitFor ("element " ++ unpack css ++ " to be removed from the page") $ check >>= expect
    where
        check =
            (maybe findElem findElemFrom from (ByCSS css) >> return False)
                `catch` \(FailedCommand ty _) -> return $ ty == NoSuchElement

Specs, Actions, and Expectations

Pending Spec

Once I am ready to start testing a page, the first thing I do is sketch out the entire spec using pending. For example, I would start with a file LoginSpec.hs with contents which look like

module LoginSpec (spec) where

import WebdriverExtended
import Test.Hspec.WebDriver

spec :: Spec
spec = do
    session "successfully logs in" $ using browsers $ do
        it "fails to submit with empty email" $ pending
        it "fails to submit with invalid email" $ pending
        it "fails to submit with empty password" $ pending
        it "enters valid email and password" $ pending
        it "logs in by pressing the button" $ pending

    session "resets a forgotten password" $ using browsers $ do
        it "clicks the forgot password link" $ pending
        it "tries to reset the password with an empty email" $ pending
        it "enters a valid email into the box" $ pending
        it "submits the reset password request and shows a message" $ pending
        it "navigates to the link from the email" $ pending
        it "enters the new password" $ pending
        it "is now logged in" $ pending
        it "logs out successfully" $ pending
        it "enters email and new password" $ pending
        it "is now logged in" $ pending
    ...

Sometimes I write this pending spec while designing the page. It helps you to think about user flow through the application. For example, you don't need to just display an error message when the password is incorrect, but provide a link or button to allow the user to reset the password. Laying out the spec as above forces you to think about sequences of pages and actions.

Next, leaving the LoginSpec.hs file with pending examples, I write a module LoginActions.hs. This is split out into separate modules because I have found that sometimes page specs use actions and expectations from multiple pages.

Actions and Expectations

LoginActions.hs will contain all the various actions that will be needed to implement the outlined session. For example, I will need an action to insert some text into the password textbox and an action to click the submit button. I have found that it is best to make everything into an action, even if it is only a single line of code, so that if there is some small refactoring that happens that perhaps changes slightly the resulting HTML DOM, it is a simple update to the action. At this stage I try and write as many actions as I think will be needed, keeping in mind the session outline in LoginSpec.hs.

To locate elements, I use the data-key property. When writing react views, elements must have keys to help the React tree-diff algorithm find matching subtrees. Ideally, since all or almost all elements have keys, it would be nice to use those keys to access elements from the test suite. Unfortunately, I couldn't find an easy way of locating elements by keys. Instead, whenever I create a key inside the react views, I also create a data-key attribute using a utility function:

key :: JSString -> [PropertyOrHandler handler]
key k = ["key" $= k, "data-key" $= k]

This passes the key to react and sets a data-key attribute on the HTML element. I can then use the data-key attributes to locate elements inside LoginActions.hs

For example,

module LoginActions where

import WebdriverExtended
import Test.Hspec.WebDriver

sendTextToLoginEmail :: Text -> WD ()
sendTextToLoginEmail t = findElem (ByCSS "input[data-key='login-email']") >>= sendKeys t

The LoginActions.hs file will also contain various expectations that the page is in some state. For example,

expectNotLoggedIn :: WD ()
expectNotLoggedIn = waitForDisplayed Nothing "form[data-key='login-form']"

expectPasswordRefreshSent :: Text -> WD ()
expectPasswordRefreshSent email = do
    e <- findElem $ ByCSS "p[data-key='top-message']"
    getText e `shouldReturn` ("We sent an email to " ++ email ++ ". Please click on the link in the email to reset your password.")

Filling in the spec

Finally, once the actions and expectations are written, it is time to fill in LoginSpec.hs and turn the specs from pending to actual code. While doing so, I resist the temptation to just use findElem, and therefore sometimes go back to LoginActions.hs to revise or expand the actions and expectations. Occassionally, it does make sense to use findElem directly from the spec when you have a one-off action.

Test Suite Development

While developing the test suite, I open the project in GHCI and can call individual actions or expectations. I only do light testing here just to make sure the findElem calls are actually finding elements (sometimes the CSS queries can be tricky).

To keep around the session across GHCI :reloads, I use the foreign-store package. Inside WebdriverExtended.hs, I have the following code:

startDevSession :: IO ()
startDevSession = do
    mstore <- lookupStore 0
    case mstore of
      Nothing -> do
        let conf = WD.useBrowser WD.chrome WD.defaultConfig
        initialS <- WD.mkSession conf
        initialC <- WD.mkCaps conf
        WD.runWD initialS $ do
            s <- WD.createSession initialC
            open "/account"
            liftIO $ writeStore (Store 0) s
      Just _ -> return ()

devR :: WD.WD a -> IO a
devR wd = do
  s <- readStore (Store 0)
  WD.runWD s wd

devSpec :: String -> Spec -> IO ()
devSpec "" s = hspec s
devSpec f s = withArgs ["-m", f] $ hspec s

I then start the production build of the servant server passing in some config settings to tell it to use the test database, serve the metalsmith static files locally, and use stack ghci acceptance-test. In the ghci window, I can use something like the following to test individual actions or expectations:

startDevSession
devR $ sendTextToLoginEmail "test@example.com"

If I find that sendTextToLoginEmail wasn't correctly finding elements because I messed up the CSS query, I can edit the code, run :reload in GHCI, and then devR $ sendTextToLoginEmail "test@example.com" again. The session will stay open across the reload because of the foreign-store package. Note that you can click around and type in the opened browser window to get the page into some state, and then switch over to GHCI and execute a few sample commands.

Finally, while developing the spec itself, I call devSpec from GHCI. This will create a new session on each call so does not need to use foreign-store.

Test Suite Execution

Once the spec has been developed, stack can be used to directly execute the test suite. A simple stack exec acceptance-test will run the entire test suite and report the results. I do this as part of continuous integration.