Chapter 6 - Acceptance Testing
By John Lenz. 2016.
Acceptance testing is black-box testing where the tests interact with the full program/web site exactly as a user does. Acceptance testing will run the production build of the server and load the metalsmith static site into a real web browser such as Firefox and Chrome. The acceptance tests then click elements on the page, send keyboard keys, and make sure the DOM is as expected. Importantly, the only interaction the acceptance test has is via the browser. Selenium and specifically hspec-webdriver make writing these kinds of tests quite easy.
What should the tests focus on? The
datastorage library and the servant server both have been extensively unit-tested so we can be reasonably confident that this code works. I am also reasonably confidant that the react-flux stores work correctly: the network communication is type-checked by the react-flux-servant library and if the store contains a complex domain data transformation, I usually move the transformation function into the
The test suite lives in an
acceptance-test subdirectory and consists of a separate cabal project. The reason for this is that the
acceptance-test cabal project does NOT depend on any server code. By making a separate cabal project, it lessens the temptation to cheat and forces the website to be tested exactly as a user will interact with the site. This
acceptance-test cabal project is built via stack and depends just on
One crucial file in the test suite is
WebdriverExtended.hs. In this file, I have some helpers for various tasks missing from the base
webdriver package. Here is a fragment:
waitFor :: String -> WD a -> WD a waitFor message w = waitUntil 10 w `catch` \e@(FailedCommand ty _) -> if ty == Timeout then error ("Waiting for " ++ message ++ " timed out") else throwM e waitForDisplayed :: Maybe Element -> Text -> WD () waitForDisplayed from css = waitFor ("element " ++ unpack css ++ " to be displayed") $ do e <- maybe findElem findElemFrom from $ ByCSS css isDisplayed e >>= expect waitForRemoved :: Maybe Element -> Text -> WD () waitForRemoved from css = waitFor ("element " ++ unpack css ++ " to be removed from the page") $ check >>= expect where check = (maybe findElem findElemFrom from (ByCSS css) >> return False) `catch` \(FailedCommand ty _) -> return $ ty == NoSuchElement
Specs, Actions, and Expectations
Once I am ready to start testing a page, the first thing I do is sketch out the entire spec using pending. For example, I would start with a file
LoginSpec.hs with contents which look like
module LoginSpec (spec) where import WebdriverExtended import Test.Hspec.WebDriver spec :: Spec spec = do session "successfully logs in" $ using browsers $ do it "fails to submit with empty email" $ pending it "fails to submit with invalid email" $ pending it "fails to submit with empty password" $ pending it "enters valid email and password" $ pending it "logs in by pressing the button" $ pending session "resets a forgotten password" $ using browsers $ do it "clicks the forgot password link" $ pending it "tries to reset the password with an empty email" $ pending it "enters a valid email into the box" $ pending it "submits the reset password request and shows a message" $ pending it "navigates to the link from the email" $ pending it "enters the new password" $ pending it "is now logged in" $ pending it "logs out successfully" $ pending it "enters email and new password" $ pending it "is now logged in" $ pending ...
Sometimes I write this pending spec while designing the page. It helps you to think about user flow through the application. For example, you don't need to just display an error message when the password is incorrect, but provide a link or button to allow the user to reset the password. Laying out the spec as above forces you to think about sequences of pages and actions.
Next, leaving the
LoginSpec.hs file with pending examples, I write a module
LoginActions.hs. This is split out into separate modules because I have found that sometimes page specs use actions and expectations from multiple pages.
Actions and Expectations
LoginActions.hs will contain all the various actions that will be needed to implement the outlined session. For example, I will need an action to insert some text into the password textbox and an action to click the submit button. I have found that it is best to make everything into an action, even if it is only a single line of code, so that if there is some small refactoring that happens that perhaps changes slightly the resulting HTML DOM, it is a simple update to the action. At this stage I try and write as many actions as I think will be needed, keeping in mind the session outline in
To locate elements, I use the
data-key property. When writing react views, elements must have
keys to help the React tree-diff algorithm find matching subtrees. Ideally, since all or almost all elements have keys, it would be nice to use those keys to access elements from the test suite. Unfortunately, I couldn't find an easy way of locating elements by keys. Instead, whenever I create a key inside the react views, I also create a
data-key attribute using a utility function:
key :: JSString -> [PropertyOrHandler handler] key k = ["key" $= k, "data-key" $= k]
This passes the key to react and sets a
data-key attribute on the HTML element. I can then use the
data-key attributes to locate elements inside
module LoginActions where import WebdriverExtended import Test.Hspec.WebDriver sendTextToLoginEmail :: Text -> WD () sendTextToLoginEmail t = findElem (ByCSS "input[data-key='login-email']") >>= sendKeys t
LoginActions.hs file will also contain various expectations that the page is in some state. For example,
expectNotLoggedIn :: WD () expectNotLoggedIn = waitForDisplayed Nothing "form[data-key='login-form']" expectPasswordRefreshSent :: Text -> WD () expectPasswordRefreshSent email = do e <- findElem $ ByCSS "p[data-key='top-message']" getText e `shouldReturn` ("We sent an email to " ++ email ++ ". Please click on the link in the email to reset your password.")
Filling in the spec
Finally, once the actions and expectations are written, it is time to fill in
LoginSpec.hs and turn the specs from pending to actual code. While doing so, I resist the temptation to just use
findElem, and therefore sometimes go back to
LoginActions.hs to revise or expand the actions and expectations. Occassionally, it does make sense to use
findElem directly from the spec when you have a one-off action.
Test Suite Development
While developing the test suite, I open the project in GHCI and can call individual actions or expectations. I only do light testing here just to make sure the
findElem calls are actually finding elements (sometimes the CSS queries can be tricky).
To keep around the session across GHCI :reloads, I use the foreign-store package. Inside
WebdriverExtended.hs, I have the following code:
startDevSession :: IO () startDevSession = do mstore <- lookupStore 0 case mstore of Nothing -> do let conf = WD.useBrowser WD.chrome WD.defaultConfig initialS <- WD.mkSession conf initialC <- WD.mkCaps conf WD.runWD initialS $ do s <- WD.createSession initialC open "/account" liftIO $ writeStore (Store 0) s Just _ -> return () devR :: WD.WD a -> IO a devR wd = do s <- readStore (Store 0) WD.runWD s wd devSpec :: String -> Spec -> IO () devSpec "" s = hspec s devSpec f s = withArgs ["-m", f] $ hspec s
I then start the production build of the servant server passing in some config settings to tell it to use the test database, serve the metalsmith static files locally, and use
stack ghci acceptance-test. In the ghci window, I can use something like the following to test individual actions or expectations:
startDevSession devR $ sendTextToLoginEmail "email@example.com"
If I find that
sendTextToLoginEmail wasn't correctly finding elements because I messed up the CSS query, I can edit the code, run
:reload in GHCI, and then
devR $ sendTextToLoginEmail "firstname.lastname@example.org" again. The session will stay open across the reload because of the foreign-store package. Note that you can click around and type in the opened browser window to get the page into some state, and then switch over to GHCI and execute a few sample commands.
Finally, while developing the spec itself, I call
devSpec from GHCI. This will create a new session on each call so does not need to use foreign-store.
Test Suite Execution
Once the spec has been developed, stack can be used to directly execute the test suite. A simple
stack exec acceptance-test will run the entire test suite and report the results. I do this as part of continuous integration.