a few key topics to consider are:
- Test-driven development (TDD) –Test driven development involves writing unit tests, watching them fail, and then implementing the source code required to make the test pass. While many organizations do not utilize TDD, I believe every developer should experience it at least once. Among other benefits, TDD makes developers contemplate the purpose of their source code before they write it.
- Non-traditional development organizations –With the rise of offshore, community, and distributed development, test automation process is even more critical. In these environments, agreements on how tests will be written and how test success will be measured should be determined before development begins. Code coverage and number of tests can also be a valuable progress indicator.
Leverage Existing Tools
The last few years have provided an explosion in the number and quality of software tools available to assist with automated testing. With such a wide array of tools available, I'll touch upon the primary families of test automation software to point readers in the right direction, but it’s important to compare test automation tools as they apply to your organization's personal needs and quality strategy. And while these tools have traditionally been separate solutions, recent products such as Microsoft® Visual Studio Team System are offering more comprehensive bundled solutions.
- Unit testing framework –This is the actual software library that allows test to be easily written and executed. Popular test frameworks include JUnit for JavaTM, and NUnit and MbUnit for .NET.
- Code coverage –A code coverage tool will execute unit tests and determine which lines of application source code are exercised ("covered"). Popular tools include Cobertura for Java and NCover for .NET.
- Build tools –This provides one "script" to build the source code, execute the tests, calculate code coverage, and various other tasks. Popular build tools include Apache Ant for Java and NAnt for .NET.
- Source code repository –This is where all source code is cataloged. Popular repositories include Subversion® and Microsoft Team Foundation Server.
- Continuous integration –These tools continuously build and test the source code from the source code repository. Combined with a build script these can provide periodic reporting of test metrics.
- Miscellaneous–The above tools represent the core for an automated testing strategy. Auxiliary tools exist to further ease the automated testing leap (such as integrated development environments, mock libraries, and code analysis tools).
Metrics lead to Measurement
Test automation metrics can help teams set goals and measure progress. Most successful test organizations track and publish test automation metrics, which inspire pride and competition among their teams. Most of the build tools mentioned above support a task to create a daily report and automatically email it to team members. As you consider what metrics to keep in order to drive measurement and analysis of testing, consider the following, most common measurements:
- Source code coverage –This is the percentage of application code that is exercised by automated tests. Most organization set a code coverage goal for all code of 80 to 90 percent.
- Total number of tests or total lines of test source code –These absolute tracking metrics such as KLOCs (bugs per thousand lines of code) can provide a measure of progress when tracked over time.
- Velocity–When the total number of tests or lines of test source code is tracked over a period of time a team can measure their "velocity"–the number of tests written per day, and/or week. This is an especially useful metric for agile teams that need to estimate work done per iteration.
- Test code to application code ratio –The ratio of the absolute lines of source code in