Central to the success of any Scrum project are the acceptance criteria. But what exactly are acceptance criteria and how do they contribute to the overall project success? In this article, we will demystify the concept of acceptance criteria and explain everything you need to know.
Acceptance criteria serve as the objective measures that define when a user story or a feature is considered complete. They outline the specific conditions that must be met for the software to be considered acceptable by stakeholders. Understanding and using acceptance criteria effectively is crucial for guiding the development team, ensuring a common understanding among stakeholders, and delivering a successful final product.
In this comprehensive guide, we will delve into the purpose and benefits of acceptance criteria, explore the best practices for writing them, and provide practical examples to illustrate their application in real-world scenarios. Whether you’re new to Scrum or an experienced practitioner, this article will empower you with the knowledge and tools to effectively define and utilize acceptance criteria in your software development projects.
User Stories and Acceptance Criteria
A user story provides the context about who the user is, what they need, and why they need it. User stories also foster a collaborative decision-making approach within the Scrum team, so that the team members define how they are going to implement or build the solution, and what the solution is. Acceptance criteria add the guardrails to establish quality and reduce ambiguity, because without acceptance criteria, the team would not know when the work is done or when the work is done properly.
The Product Owner needs to define these acceptance criteria, and typically they do this as part of the collaboration with the Developers that takes place during the refinement activities. As the team refines the product backlog and the user stories, the acceptance criteria get added, clarified, and refined, so that everyone on the team can be on the same page.
What Acceptance Criteria are
Acceptance criteria represent a list of predefined conditions or requirements that must be met so that the work on the story can be considered done. Acceptance Criteria include standards of quality established by the organization and by the team.
When the Developers work on a story, they need to ensure their work satisfies all the acceptance criteria defined for that story. If any criterion is not true, then the work on the story cannot be considered completed, and the Developers need to improve or fix it until all criteria are satisfied.
Microsoft defines Acceptance Criteria as, “Conditions that the software product must satisfy to be accepted by the user, customer, or other stakeholder.”
Google defines Acceptance Criteria as, “Pre-established standards or requirements a product or a project must meet.”
Below is an example of a user story. Imagine that we are working for a travel website like Expedia or an airline, and we want to build a new functionality. The user story describes it:
“As an international traveler waiting in the airport, I want to receive an alert when it’s time to board my plane so that I can do shopping in the airpo2rt while waiting and I don’t risk losing my flight.”
The user story describes the functionality that we need to build into the system, and it does so from the user’s point of view. It specifies what the user wants to do and what benefit it would get from that functionality. But if you are a Developer, how do you know when your work on that story is completed and is done properly? How do you know when you have built this functionality in the right way, as expected by the user?
You need to establish some acceptance criteria.
For example: “We want the alert to be sent via text message to the number on file”. This criterion establishes the user experience that we want to deliver. We want to make sure that we send the alert via text message and not, for example, via email. In fact, if the Developers send it via email, it will not pass this acceptance criteria and we need to fix the system.
Another example: “We want to send it via text message on the number that we have on file for the customer.” Not a random number, but the specific number of file for that specific customer.
And: “The alert is sent between 15 to 25 five minutes before boarding, so that we give enough time to the passenger to come to the gate.” If we send the alert only 5 minutes before boarding, it may not give the user enough time to reach the gate.
Where Acceptance Criteria come from
Notice that once we have a list of criteria, we can test if the functionality satisfies them and if the work is actually done to the quality levels expected. You may call these requirements.
You may also say that these criteria describe the type of customer experience that we want to deliver to the customer. Typically, the Product Owner (or the Business Analyst if you have one) defines these acceptance criteria. This helps the Product Owner be clear on the quality level and the user experience that they want to deliver to the end user.
However, some criteria could be technical in nature, for example if you want to ensure a specific level of performance or security in the system. Maybe the Product Owner is not technical enough to define these criteria. So, where do these (more technical) criteria come from? They come from the Developers.
During refinement, as the Product Owner and the Developers collaborate at refining a specific story, they can add additional criteria. The Developers may even suggest specific criteria to strengthen the performance, the security, or in general the quality of the work they will deliver.
So, while the Product Owner is responsible for making sure the acceptance criteria for a user story are defined, they don’t need to do all the work alone. They can collaborate with the Developers (and maybe the Business Analyst) to define them.
Testing the Acceptance Criteria
Once you have a user story with acceptance criteria, the Developers can pull it into a sprint and do the work. As the Developers build this functionality, as they work through this story, they will test their work and they will make sure that all of the acceptance criteria for that story are satisfied.
If one or more of these criteria are not satisfied, the story is not done. In fact, some of the tests will fail and the work goes back to the Developers as they need to fix it. When they fix it and then finally pass all the tests, we can say that all the acceptance criteria have been validated. The work is completed and story is done.
For this process to work, the testing should be performed throughout the sprint. If something does not work or any acceptance criteria is not satisfied, the Developers can fix and complete the work within the sprint. This ensures that the work that the Developers selected in the Sprint Backlog is actually completed 100% by the end of the Sprint.
Instead, if the Developers do the testing at the end of the Sprint and they discover that a test fails or an acceptance criterion is not satisfied, now they don’t have any time to fix what’s missing. The sprint is over, and the work is not done.
Types of acceptance criteria
We can divide the acceptance criteria in 4 different categories.
Functional requirements: Acceptance criteria define the functional requirements for the intended user experience. They describe what the system should do and how it should behave to deliver the right customer experience to the end users.
Test conditions: In fact, all of the acceptance criteria should be tested so that you can check whether you are delivering the work in the right way or not, and with the right level of quality. Acceptance criteria should be defined in terms of pass or fail scenarios: once we run the test, we should be able to clearly state if the criteria have been satisfied or not, with no “gray area” interpretation.
Negative scenarios: The acceptance criteria should also define the negative scenarios. It is useful to describe how the system needs to behave when things go wrong or in a way that is unexpected. Sometimes, negative scenarios can add complexity to the work required to build a new functionality, and you may want to be clear about what needs to happen. For example, if you are building a credit card payment page, what should the system do if the user enters the wrong credit card number? What action should the system take and what error message should the user see? What should the system not do in this situation?
Non-functional requirements (NFR): These may include criteria that define the expected level of performance, of security encryption, or of integration with other systems. These are things that the end user may not directly know about or may not experience, but that we consider important for the level of quality of our product. For example, if you are building an app for a bank, very likely you need to ensure all data is encrypted for security and privacy purposes. So, every story should have an NFR specified in the acceptance criteria. By doing this, you set expectations about what constitutes good quality for the work you are doing, and you create conditions that can be tested.
One last comment about acceptance criteria: Acceptance criteria do not describe implementation details, so it’s not about writing down a long list of tasks or describing how we are going to do the work. These implementation details don’t need to go into your story. The implementation details are discussed and agreed upon by the Developers, and they can be defined in a separate document that is attached to the user story.
The how we build a piece of functionality, how we implement a story, what kind of database integration, API, UI design layout… All of these things are not part of the user story. They can be added as a separate document (an implementation details document). In fact, most of the tools that we use today, Jira, Trello and so on, they allow you to attach a document to each of the stories or add a link to a Confluence page. That’s where the Developers put all the implementation details.
Writing good Acceptance Criteria
How do you know when the acceptance criteria are properly written?
To start, the acceptance criteria should be testable, because they provide the “Confirmation” part in the user story. So, they should be written in a way that we can test them, and when we test them, the end result should be either a pass or a fail, and if it’s a fail, it needs to be fixed by the Developers. In fact, that’s the main reason why we write them in the first place: to provide quality guardrails to the Developers.
The criteria should be clear and concise, so it’s clear what it is that we need to test. Everyone must understand the criteria. They can include both functional and non-functional requirements.
Because you can write them in different ways, we are going to look at 3 different methods to write the acceptance criteria.
Let’s consider this story:
“When I am ready to complete an online purchase on the website, I want to pay with a credit card so that I can complete the checkout quickly.”
This is a job story to allow people to pay with a credit card, for example on an e-commerce website. Now, to build this story, we need a list of acceptance criteria.
1. Things we need to test
The first method to describe the acceptance criteria is to create a list of all the things that we need to test.
- The address inserted by the user needs to match the billing address on file for the card. We need to test that the system is checking for that and that the two addresses actually match.
- The CVV (security code) matches.
- The expiration date entered by the user matches the one on the card.
- We support Visa, MasterCard, but we do not support American Express, so we want to make sure that we test and we check which card has been entered. And if it’s the wrong card we provide an error message and stop the processing.
In this first method, we just create a list of acceptance criteria, and we are going to use all of them for testing purposes.
2. List of scenarios
The second method is to explicitly provide a list of scenarios and the expected behavior to validate. For each scenario, you define the expected result of that scenario, and if the system behaves in a different way it fails.
For example, when the name on the card, the address, the CVV, and the expiration date all match, in this scenario the expected behavior of the system is to pass, to accept the payment.
- If the address does not match the billing address, the expected behavior is to fail and provide an error to the user.
- If the card type equals MasterCard or Visa, the expected behavior should be to pass and accept the payment. Instead, if the card type is American Express, the expected behavior should be to provide an error to the user, and to fail the test.
3. Gherkin format
There is a third method that we can use to represent acceptance criteria, and that is the Gherkin format. The Gherkin format is a format that you can use to write acceptance criteria. Here is the structure:
Given [One or more conditions]
When [An event occurs or action is taken]
This describes what the system should be doing when something happens under a specific condition. For example:
Given the customer enters a credit card
When the security code does not match the CVV
Then stop and show an error to the user
What is the value of using the Gherkin format? It is valuable for several reasons, but the main one is that the Gherkin format is a human readable, machine interpretable language. It means that humans understand what we are saying and a machine can interpret the language and automate the tests.
There are platforms like Cucumber or Selenium (and others) that are able to read this language and perform the testing. When this is happening, you don’t need manual testing anymore. You can automate all the testing, and you start entering in TDD territory (Test Driven Development), or even in regression testing territory.
When you’re able to automatically test all the prior tests, all the prior criteria from even 2 years ago, you make sure that when you make a change to your system, nothing from the past breaks. If you do that manually, it’s going to take forever, it’s impossible to do. But when you automate that, a regression testing capability really brings the stability of the system up several notches. It improves the ability of the team to develop, and maintain, a quality product.
You may also enter into DevOps territory, and a lot of teams in software development are adopting DevOps practices, TDD, regression testing, and so on. These are good practices because they allow the team to improve the quality of the product and as a result also improve its performance.
Difference between acceptance criteria and definition of done
At a high level, they represent the same thing: They both represent the criteria that need to be satisfied for the work performed by the Developers to be considered completed to the level of quality defined for the product. However, we use the different terminology – acceptance criteria and definition of done – to distinguish between criteria defined for each individual PBI and criteria that are true for all PBIs.
Acceptance criteria are defined specifically for one PBI or for one user story. Each story has its own set of acceptance criteria. All of the acceptance criteria for a given story need to be satisfied for that story to be done.
The definition of done is defined at the team level and it represents a list of criteria that apply to everything that the team does, so every single story and PBI.
When used together, the acceptance criteria and the definition of done create a powerful set of quality guardrails for the work performed by the team to be considered both done and high-quality. In practice, every story or PBI selected for a sprint needs to satisfy both the acceptance criteria defined for the story, and the criteria in the definition of done.
One way to think about this is that acceptance criteria define the criteria that are specific to each individual story. The definition of done defines the criteria that are common to all the work that you are doing, so you don’t need to repeat those criteria in every single story, you just put them in the definition of done.
The definition of done can also include standards of quality that the team wants to ensure for how the team members work together. For example, “No story should be marked completed, unless the team has 1) completed peer review and 2) the Product Owner has reviewed and accepted the work result.”
Another example, “Everything that we do must go through security review and be approved before we release it to production.”
These examples may originate in the working agreement between team members. And by putting these criteria in the definition of done, the team can ensure a strict adherence to them.
For example, when I was at Capital One every single thing that we did had to go through a security review. This makes sense because Capital One is a bank and of course security is key to any financial institution.
Even if we wanted to change just photo on the app, or if we wanted to change a line of text on the webpage, well…. security review. Whatever we did had to go to security review.
Looking back to our definition of done, we had a criterion called “Confirm pass of security review”, and if any work item did not pass it, our work was not done, and we had to fix it. This criteria was defined once in our definition of done, and applied to every individual story the team worked on.