Acceptance testing at synyx – Part 5

The last few blogs about acceptance-testing focused on setting up a nice and scalable infrastructure to do testing through the (web)-GUI using a Selenium grid. Since we’ve got this running now we can go on to topics that focus how we write these tests. At synyx we try to write our web-tests as “acceptance-tests” so we first take a small dive into what that is.

What is an Acceptance Test?

In the first place an acceptance test cares about what is tested, not so much about how this is done. Consider that you specify features for an application, hence you write user stories for them. Then you will soon get to the question, when the work at a story is completed. A good approach to define when you are really done with a story is to define some acceptance criteria for it. Then you check if the application meets these criteria to determine if the story you are working on is done. So if a story has the title “A shop-item can be added to the shopping cart” you probably can define acceptance criterias pretty easy. One of them could be “each item in the shop that is currently available for ordering can be added to the users shopping cart. If the user views his cart afterwards the item is listed there”. Traditionally these criteria might get tested by the developers themself and later some variation of QA. But it’s preferable if these criteria are tested automatically somewhere in the build pipeline.
Automating acceptance-testing has big advantages over manual testing: Noone has to do the same thing all over again. Because if a human does, he will make mistakes. He will forget to test something and maybe skip over other things. Also, manual tests are boring and take alot of time. And if you get more and more tests by the time you’d have to hire more and more people which will cost alot of money. Much more than simply scaling your test-cluster up.
But lets think about this… If you are writing unit tests you probably also write acceptance tests. Yes, some tests are rather technical and low-level. Therefore they will not be part of the acceptance criteria of a user story given to the developers by the product owner. But some of the tests we write are acceptance-tests “by accident”:

public void addsAvailableItemsToCart() {
  Item item = new Item("synyx coffee mug");
  Assert.assertThat(cart.getItems(), containsItem(item));
@Test(expected = ItemUnavailableException.class)
public void doesNotAddUnavailableItemsToCart() {
  Item item = new Item("synyx coffee mug");
  cart.add(item);  // exception expected

As you can see the two test methods above could be a way to verify the criteria of the story “A shop-item can be added to the shopping cart” I described above.
But – of course – there are other ways to test the same thing. The test above looks like a unit test (our unit is the shopping cart implementation We could also test on service-level or through the GUI. Since the mentioned example is kind of trivial there is no need to test it on another level than the unit-level. But some stories have more complex acceptance criteria and need more complex tests that have to be tested on higher levels. You might also want to be sure that some components work together to complete a task and you want to test these together.
Acceptance criteria should focus on functional requirements and therefore be without technical details. The criteria belongs directly to the user stories and should be created along with the user stories. Because requirements or stories are usually written by non-technical pepole a common language is needed: Business-People have to be able to write them and developers or testers have to understand and implement them. This is why I prefer the BDD-Style of specifig acceptance criteria. Here you define usage scenarios of the application / the story. The scenario defines what the user wants to do, how this breaks down to single steps, what preconditons have to be met and what the expected outcome of the actions are. So these criteria are often verbalized in given/when/then form:

Given I look at the details of the item "synyx coffee mug"
When I add the item to the shopping cart
Then there is a "synyx coffee mug" in my Cart.

As you can see I wrote the acceptance criteria based on the GUI my webshop has. I think this is much easier for a non-technical person since noone has to focus on services, components and whatever exists in your webshop application but on what the product owner “knows” and sees (or wants to see) when he is using the application.
This is the time it comes in handy to have a selenium-grid at our service. So in the next few posts we are gonna discuss how to write tests using selenium to do acceptance testing through the GUI. And – of course – we also gonna discuss how to turn the BDD-Style criteria into code. So again… stay tuned 🙂


  1. JBehave, baby!