Wednesday, July 14, 2004
Defining Tasks with Acceptance Tests
Here's an email from a few years back:
From: Reg Braithwaite-Lee
Sent: Monday, August 28, 2000 6:52 PM
To: XXX
Subject: RE: Software
XXX wrote:
The terse response is that you must be synchronized on expectations.
My practise is to set up "acceptance tests." An acceptance test is a written understanding of what it means for a piece of software or task to be "done." An acceptance test has a Boolean output: pass/fail, done/not done.
Acceptance tests start with a one sentence executive summary. The best acceptance tests strive to be objective "The sort function shall be deemed to have acceptable performance when it can demonstrate sorting 100,000 randomly generated strings in x seconds or less."
When you and the developer have agreed on a one sentence summary, you drill down to specific tests. You can probably dictate the summary, but the developer should participate as a reasonably equal partner in fleshing out the tests:
"The sort function shall be deemed to have acceptable performance when it can demonstrate sorting 100,000 randomly generated strings in x seconds or less:
Random strings to be in a text file, one string per line, strings to be 10 characters long. Standard Unix conventions. Test to be run using a simple shell that reads the entire file into RAM before timing the sort. Timing will be established using the systemTimeInMilliseconds() function. The test is a pass if the time for the sort is less than x milliseconds. The machine for the testing will be any of our usual testing systems (currently YYY running Linux)."
Large point: I forbid coding until the acceptance test has been signed off. Moving ahead is tempting for the developer and the manager, but resist! I also do not accept any time estimates for a task that does not have acceptance tests defined. How does (s)he know how long it will take if you haven't agreed on when it will be done?
Small point: for some tasks, setting up and writing the tests takes as long as the coding. This is normal, and an acceptable cost of development. It is a false saving to thug on ahead to avoid the "overhead" of acceptance tests.
Medium Point: Tugging over acceptance tests is an important part of the development process: it is a valuable brainstorming activity and quickly identifies what is important and what is not.
p.s. Something like a sort requires multiple acceptance tests. Here's another for the sort function:
"The sort function shall sort strings into alphabetical order:
Random strings to be in a text file, one string per line, strings to be 10 characters long. Standard Unix conventions. A sorted copy of the file will be generated using Unix standard sort. Test to be run using a simple shell that reads the entire file into RAM and writes the sorted result to a file. The veracity of the sort will be validated by comparing the output with the Unix sort version. The machine for the testing will be any of our usual testing systems (currently YYY running Linux)."
Looking back, I didn't really address my friend's concerns directly. First, there was the concern of 'his spec', as in the developer's spec. That's really what this email addresses: making sure that the developer is working on the right thing.
There's another entire problem of 'being left alone', also known as 'going dark.' My email doesn't address that problem at all. Perhaps I'll post something on that topic in the future...
From: Reg Braithwaite-Lee
Sent: Monday, August 28, 2000 6:52 PM
To: XXX
Subject: RE: Software
XXX wrote:
I have a software guy who is undisciplined - he wants to just be left alone and write software to his spec.
Any tips and hints there for me? :)
The terse response is that you must be synchronized on expectations.
My practise is to set up "acceptance tests." An acceptance test is a written understanding of what it means for a piece of software or task to be "done." An acceptance test has a Boolean output: pass/fail, done/not done.
Acceptance tests start with a one sentence executive summary. The best acceptance tests strive to be objective "The sort function shall be deemed to have acceptable performance when it can demonstrate sorting 100,000 randomly generated strings in x seconds or less."
When you and the developer have agreed on a one sentence summary, you drill down to specific tests. You can probably dictate the summary, but the developer should participate as a reasonably equal partner in fleshing out the tests:
"The sort function shall be deemed to have acceptable performance when it can demonstrate sorting 100,000 randomly generated strings in x seconds or less:
Random strings to be in a text file, one string per line, strings to be 10 characters long. Standard Unix conventions. Test to be run using a simple shell that reads the entire file into RAM before timing the sort. Timing will be established using the systemTimeInMilliseconds() function. The test is a pass if the time for the sort is less than x milliseconds. The machine for the testing will be any of our usual testing systems (currently YYY running Linux)."
Large point: I forbid coding until the acceptance test has been signed off. Moving ahead is tempting for the developer and the manager, but resist! I also do not accept any time estimates for a task that does not have acceptance tests defined. How does (s)he know how long it will take if you haven't agreed on when it will be done?
Small point: for some tasks, setting up and writing the tests takes as long as the coding. This is normal, and an acceptable cost of development. It is a false saving to thug on ahead to avoid the "overhead" of acceptance tests.
Medium Point: Tugging over acceptance tests is an important part of the development process: it is a valuable brainstorming activity and quickly identifies what is important and what is not.
p.s. Something like a sort requires multiple acceptance tests. Here's another for the sort function:
"The sort function shall sort strings into alphabetical order:
Random strings to be in a text file, one string per line, strings to be 10 characters long. Standard Unix conventions. A sorted copy of the file will be generated using Unix standard sort. Test to be run using a simple shell that reads the entire file into RAM and writes the sorted result to a file. The veracity of the sort will be validated by comparing the output with the Unix sort version. The machine for the testing will be any of our usual testing systems (currently YYY running Linux)."
Looking back, I didn't really address my friend's concerns directly. First, there was the concern of 'his spec', as in the developer's spec. That's really what this email addresses: making sure that the developer is working on the right thing.
There's another entire problem of 'being left alone', also known as 'going dark.' My email doesn't address that problem at all. Perhaps I'll post something on that topic in the future...
Labels: agile
¶ 8:21 AM
Did you enjoy this post? You can find more like it at weblog.raganwald.com. Better yet, subscribe in a reader, and you'll get my essays and links of interest to software developers, with no extra charge for delivery! Did I mention that I love coffee? If you would like to “fuel” my writing, please consider buying me a large black coffee, no sugar, for just $1.47. Thanks!
Reg,
Your comments are right on the mark. Much of this is literally programming 101. Break a problem down into its component parts and then attack each one. The solution for each part either works or it doesn't. If all the parts of a solution work - chances are the overall outcome is also possitive.
Setting the pass/fail criteria is critical.
The problem is that often times programmers forget this phase in programming process. They've 'seen it all before' and 'know what their doing' and would rather 'start hacking' then take the time to really define the problem and define their plan of attack - and define how they know a task is done.
andre
Post a Comment
Your comments are right on the mark. Much of this is literally programming 101. Break a problem down into its component parts and then attack each one. The solution for each part either works or it doesn't. If all the parts of a solution work - chances are the overall outcome is also possitive.
Setting the pass/fail criteria is critical.
The problem is that often times programmers forget this phase in programming process. They've 'seen it all before' and 'know what their doing' and would rather 'start hacking' then take the time to really define the problem and define their plan of attack - and define how they know a task is done.
andre
# posted by 
: 5:56 PM
: 5:56 PM<< Home
