An ex-colleague of mine was telling me about a situation that he experienced on a project recently whereby his team had written a comprehensive unit test suite for a component designed to generate XML files for consumption in another system. Their code passed all the unit tests, yet when they delivered the component to the client system, it transpired that their component was not generating the XML in the correct format.
We spoke about the benefits and costs of unit testing and ended up discussing the value of unit testing – after all, his developers had written a bunch of unit tests, but the project still had issues when running against the final system – issues that it was felt should have been identified earlier.
I was of the opinion that unit tests were being misused here; something did not sound quite right to me. I felt acceptance tests were required rather than unit tests. In order to explain why I felt this, let’s define what unit and acceptances tests generally are – and what they are not.
Defining Unit Tests
What are unit tests for? They serve several purposes e.g. allowing refactoring with confidence, regression testing, protecting code from violating SRP, providing developer-level documentation etc.. Note that nowhere here do we mention anything about unit tests defining requirements. How could they? The tests are most likely written by the developer who writes the code.
Who are unit tests for? The developers. They are there to give the developer confidence that what they think they have written is what they really have written.
Unit tests are not there for a BA or PM to say “all the unit tests are green – therefore the requirements have been met”. Never fall into this trap. It is easy to make, and for simple components, or for ones where the problem space is easily defined, you may be able to get any without anything more. But once you get into complex systems, or areas that are difficult to prove by simple unit tests, you need something more.
What are acceptance tests for? They protect against the problem of having “fuzzy” requirements. They force the authors of the test to really think about what they expect to happen in a given situation. They should also provide a form of regression suite that is ideally available to the whole team.
Who are acceptance tests for? The business. They give them confidence that the system delivered does what they asked. As a secondary beneficiary, it should also serve the developers to provide them with guidance on what the expectations from the business are.
Let me be clear: Never have a developer write acceptance tests. Whether this acceptance test is materialised by a C# unit test, or Fitnesse or Cucumber is irrelevant. The main point is that they are delivered by someone outside of the developers writing the code, and represent a contract that must be fulfilled by the developer.
This post is probably beginning to sound a bit like software development 101, but you’d be surprised the amount of times that this part is left out of the development life cycle, sometimes with extremely costly results. The only way you can truly have confidence in your ability to fulfil requirements is to code against acceptance tests that are written (ideally) by the business, or at least a business analyst, that have a good understand of what is needed. These tests run repeatedly against your code and give everyone confidence that what you are writing is what is actually required.
Think back to the scenario that I illustrated at the start of this post. Why had this situation occurred? Because unit tests were being treated as a form of acceptance test. This is something that unit tests should never be used for. It’s dangerous because it gives a false sense of security to everyone involved. I’m a big fan of unit testing, and have found designing and developing software much more enjoyable since using TDD as a day-to-day practice. But at the same time, I’d hate it to unfairly come under criticism for not providing something it doesn’t claim to.
Unit Tests prove that the system does what the developer thinks it should do
Acceptance Tests prove that the system does what the client needs it to do