(This is a snapshot of my old weblog. New posts and selected republished essays can be found at raganwald.com.)
Friday, July 30, 2004
How to Use Acceptance Tests to Ship Software
Validation tests are tests that measure some aspect of an implementation's behaviour against an expectation and answer whether the software complies with the expectation. For example, a test that measures the time required to perform a specific task is not a validation test. A test that measures the time required to perform a specific task and compares it to a benchmark is a validation test.
One very strong benefit of using validation tests is that creating the tests forces the development team to achieve detailed and objective designs. This extends beyond the coding team to include program and product management: writing tests that validate compliance with requirements force product management to articulate detailed and objective requirements.
Thinking the requirements through in detail is more valuable than thinking the design through in detail: if the design is not solid, you will end up with a poor solution to a good problem. This can be refactored. If the requirement are not well understood, you will end up with a solution to the wrong problem: this is more difficult to change.
The two most important types of validation tests are:
Acceptance tests validate an implementation's compliance with requirements. One style of development is to write detailed requirements and then translate the requirements into acceptance tests. When the requirements are sufficiently detailed, the acceptance tests flow freely from the requirements and the role of the team is to validate that the acceptance tests properly express the requirements.
A lightweight approach is to write less detailed requirements and to consider the acceptance tests as an extension of the requirements document. This works well in smaller organizations. When using the lightweight approach, it is important that customer or market facing contributors retain 'ownership' of the expression of requirements. A protocol I have seen work very well is for a program manager to write requirements, and then the program manager negotiates acceptance tests with the programming team.
The program manager owns the definition of the acceptance tests, and the programming team validates the tests, raising red flags if they have concerns or cannot 'connect the dots' between the requirements and the acceptance tests. For example, a programmer is allowed to reject any acceptance test that validates an implementation rather than a requirement. This is quite properly the province of unit tests and not an issue for the program manager to determine.
Let's review this last point. Consider a high-level Market Requirements Document ("MRD"). What does it mean to "flesh out the MRD with more detail" or to "drill down into one of the requirements"? Well, it doesn't mean "write specs for a solution." That's a common pitfall: when asked for more detail, product managers get sucked into designing software.
More detail for an MRD should mean "describe the problem to be solved in more detail." The most useful detail possible is a complete list of acceptance tests.
Writing an acceptance test is easy: write a sentence that begins: "the software is acceptable when..." Now all you have to do is make sure that your acceptance test is implementation-free, just like any good requirement. This test is now a detailed expansion of some requirement. Write as many acceptance tests as you can.
For incredible leverage, automate your acceptance tests using tools like Fitnesse.
Unit Tests
Unit tests validate an implementation's compliance with a design. Since requirements are implementation-free, there should be no coupling between the definition of unit tests and the definition of acceptance tests.
Unit tests express one and only one fact, which they express in the abstract: code that is written with well chosen and sufficient unit tests can be considered of good quality and likely to be delivered on time with high confidence. Customer or market facing contributors such as program managers have an abstract interest in unit tests for this reason.
Why Unit Test are not Acceptance Tests
Unit tests should not be confused with acceptance tests, for the same reasons that
Unit tests should not be confused with acceptance tests because they do not express some requirement of the release plan: it is possible to completely refactor the design and the project plan, rewriting every single unit test, without changing any of the acceptance tests. But changes to acceptance tests reflect changes to the product contract: they are an element of the Plan of Record.
Bugs and Defects
When a release (internal or external) is tested, bugs are inevitable. A bug report expresses a kind of validation test in the negative: "the software ought to... But instead it..." A good bug report explains how to reproduce the problem: "Install the software on a dual Triton system with less an 4GB of RAM, then boot it in greedy mode..."
This is a validation test. Ideally, this test was already expressed as an acceptance or unit test. In practice, a bug report often expresses an implicit test, one that wasn't spelled out as part of the process. This is not a bad thing: tests evolve as the team's understanding evolves.
It's important to distinguish bugs that reflect shortcomings in acceptance from bugs that reflect shortcomings in design. If you look at a bug and it reflects some fundamental aspect of the requirements of the software, you need to treat it with considerably more care than a bug that reflects some shortcoming in design.
I call such bugs "defects." A defect must absolutely, positively be remedied, because if the software does not conform to the acceptance criteria, it is not "done."
For example, you may have a requirement that some aspect of the UI conform to generally accepted principles. Say there's a menu option that displays a shortcut key, but the key doesn't work. That bug is interesting. You can write a unit test for this bug: "pressing control J should invoke the Jump command."
But that isn't an acceptance test. An acceptance test would be "for every menu item that displays a shortcut key, the shortcut key should invoke the command." Why the generalization? Because one way to fix the broken control J would be to not display the shortcut key in the menu. The software would still conform to the acceptance test, but it would fail the unit test.
Bug reports are an excellent source of acceptance criteria, however product management and development need to analyze the tests to decide which bugs are defects and to extract acceptance criteria, just as requirements need to be analyzed to remove implementation, design, and specifications.
I'll end with one anti-pattern: if you don't distinguish bugs that reflect acceptance shortfalls from bugs that reflect implementation shortfalls, you paint yourself into a corner where you cannot manage the bug list: all you can do is try to fix as much as you can and hope for the best.
If you separate the defects out and give them their own special place in your process, you are now (a) learning more about the detailed requirements of your software, and (b) you have a smaller list of defects that must absolutely, positively be fixed. Prioritizing defects over bugs is one excellent way to raise your chances of shipping acceptable software on time.
Validation tests are tests that measure some aspect of an implementation's behaviour against an expectation and answer whether the software complies with the expectation. For example, a test that measures the time required to perform a specific task is not a validation test. A test that measures the time required to perform a specific task and compares it to a benchmark is a validation test.
One very strong benefit of using validation tests is that creating the tests forces the development team to achieve detailed and objective designs. This extends beyond the coding team to include program and product management: writing tests that validate compliance with requirements force product management to articulate detailed and objective requirements.
Thinking the requirements through in detail is more valuable than thinking the design through in detail: if the design is not solid, you will end up with a poor solution to a good problem. This can be refactored. If the requirement are not well understood, you will end up with a solution to the wrong problem: this is more difficult to change.
The two most important types of validation tests are:
- Acceptance tests, which validate an implementation's compliance with requirements.
- Unit tests, which validate an implementation's compliance with a design.
Acceptance tests validate an implementation's compliance with requirements. One style of development is to write detailed requirements and then translate the requirements into acceptance tests. When the requirements are sufficiently detailed, the acceptance tests flow freely from the requirements and the role of the team is to validate that the acceptance tests properly express the requirements.
A lightweight approach is to write less detailed requirements and to consider the acceptance tests as an extension of the requirements document. This works well in smaller organizations. When using the lightweight approach, it is important that customer or market facing contributors retain 'ownership' of the expression of requirements. A protocol I have seen work very well is for a program manager to write requirements, and then the program manager negotiates acceptance tests with the programming team.
The program manager owns the definition of the acceptance tests, and the programming team validates the tests, raising red flags if they have concerns or cannot 'connect the dots' between the requirements and the acceptance tests. For example, a programmer is allowed to reject any acceptance test that validates an implementation rather than a requirement. This is quite properly the province of unit tests and not an issue for the program manager to determine.
Let's review this last point. Consider a high-level Market Requirements Document ("MRD"). What does it mean to "flesh out the MRD with more detail" or to "drill down into one of the requirements"? Well, it doesn't mean "write specs for a solution." That's a common pitfall: when asked for more detail, product managers get sucked into designing software.
More detail for an MRD should mean "describe the problem to be solved in more detail." The most useful detail possible is a complete list of acceptance tests.
Writing an acceptance test is easy: write a sentence that begins: "the software is acceptable when..." Now all you have to do is make sure that your acceptance test is implementation-free, just like any good requirement. This test is now a detailed expansion of some requirement. Write as many acceptance tests as you can.
For incredible leverage, automate your acceptance tests using tools like Fitnesse.
Unit Tests
Unit tests validate an implementation's compliance with a design. Since requirements are implementation-free, there should be no coupling between the definition of unit tests and the definition of acceptance tests.
Unit tests express one and only one fact, which they express in the abstract: code that is written with well chosen and sufficient unit tests can be considered of good quality and likely to be delivered on time with high confidence. Customer or market facing contributors such as program managers have an abstract interest in unit tests for this reason.
Why Unit Test are not Acceptance Tests
Unit tests should not be confused with acceptance tests, for the same reasons that
- the design should not be confused with the requirements
- the project plan should not be confused with the release plan, and,
- technical management should not be confused with product management
Unit tests should not be confused with acceptance tests because they do not express some requirement of the release plan: it is possible to completely refactor the design and the project plan, rewriting every single unit test, without changing any of the acceptance tests. But changes to acceptance tests reflect changes to the product contract: they are an element of the Plan of Record.
Bugs and Defects
When a release (internal or external) is tested, bugs are inevitable. A bug report expresses a kind of validation test in the negative: "the software ought to... But instead it..." A good bug report explains how to reproduce the problem: "Install the software on a dual Triton system with less an 4GB of RAM, then boot it in greedy mode..."
This is a validation test. Ideally, this test was already expressed as an acceptance or unit test. In practice, a bug report often expresses an implicit test, one that wasn't spelled out as part of the process. This is not a bad thing: tests evolve as the team's understanding evolves.
It's important to distinguish bugs that reflect shortcomings in acceptance from bugs that reflect shortcomings in design. If you look at a bug and it reflects some fundamental aspect of the requirements of the software, you need to treat it with considerably more care than a bug that reflects some shortcoming in design.
I call such bugs "defects." A defect must absolutely, positively be remedied, because if the software does not conform to the acceptance criteria, it is not "done."
For example, you may have a requirement that some aspect of the UI conform to generally accepted principles. Say there's a menu option that displays a shortcut key, but the key doesn't work. That bug is interesting. You can write a unit test for this bug: "pressing control J should invoke the Jump command."
But that isn't an acceptance test. An acceptance test would be "for every menu item that displays a shortcut key, the shortcut key should invoke the command." Why the generalization? Because one way to fix the broken control J would be to not display the shortcut key in the menu. The software would still conform to the acceptance test, but it would fail the unit test.
Bug reports are an excellent source of acceptance criteria, however product management and development need to analyze the tests to decide which bugs are defects and to extract acceptance criteria, just as requirements need to be analyzed to remove implementation, design, and specifications.
I'll end with one anti-pattern: if you don't distinguish bugs that reflect acceptance shortfalls from bugs that reflect implementation shortfalls, you paint yourself into a corner where you cannot manage the bug list: all you can do is try to fix as much as you can and hope for the best.
If you separate the defects out and give them their own special place in your process, you are now (a) learning more about the detailed requirements of your software, and (b) you have a smaller list of defects that must absolutely, positively be fixed. Prioritizing defects over bugs is one excellent way to raise your chances of shipping acceptable software on time.
Labels: agile
