and second, how long it will take to test and fix the bugs in a product that has been in development for six months and is now ready for its first round of testing. Most people will agree that estimating the new work is far easier and more likely to be accurate. Periodic (or better yet, continuous) testing of a product is a probe into that product that lets us know how far along we are.
Feedback opportunities are lost. An obvious benefit of using Scrum is that the team can get feedback on what it’s built at least at the end of every sprint. The product can be deployed onto restricted-access servers or made available for download to select customers. If the product is at a sufficient quality level for doing this only near the end of a release cycle, the team misses great opportunities to gain valuable feedback earlier.
Testing is more likely to be cut. Because of deadline pressure, work that is planned to happen at the end of a project is more likely to be dropped or reduced.
The Test Automation Pyramid
Even before the ascendancy of agile methodologies like Scrum, we knew we should automate our tests. But we didn’t. Automated tests were considered expensive to write and were often written months, or in some cases years, after a feature had been programmed. One reason teams found it difficult to write tests sooner was because they were automating at the wrong level. An effective test automation strategy calls for automating tests at three different levels, as shown in Figure 1, which depicts the test automation pyramid .
At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid. Automated unit tests are wonderful because they give specific data to a programmer—there is a bug and it’s on line 47. Programmers have learned that the bug may really be on line 51 or 42, but it’s much nicer to have an automated unit test narrow it down than it is to have a tester say, “There’s a bug in how you’re retrieving member records from the database,” which might represent 1,000 or more lines of code. Also, because unit tests are usually written in the same language as the system, programmers are often most comfortable writing them. Let’s skip for a moment the middle of the test automation pyramid and jump right to the top; the user interface level. Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible. We want this because user interface tests often have the following negative attributes:
- Brittle. A small change in the user interface can break many tests. When this is repeated many times over the course of a project, teams simply give up and stop correcting tests every time the user interface changes.
- Expensive to write. A quick capture-and-playback approach to recording user interface tests can work, but tests recorded this way are usually the most brittle. Writing a good user interface test that will remain useful and valid takes time.
- Time consuming. Tests run through the user interface often take a long time to run. I’ve seen numerous teams with impressive suites of automated user interface tests that take so long to run they cannot be run every night, much less multiple times per day.
Suppose we wish to test a very simple calculator that allows a user to enter two