Robert details how to increase the quality and scope of your automated test harness by making its development the responsibility of your development team rather than your test team.
How many times have you worked on a project, where 'all' the tests were going to be automated? Sound familiar? I am going to guess that for many of these projects the automated test effort was dropped because it took up too much time, and the test harness was too brittle to withstand even the slightest change to the application.
In the following document I aim to provide a solution to the above problem which places the responsibility of automated testing with the developer. There are a number of reasons for this, and I will detail them in the following paragraphs.
Common Problems faced when Automating Functional Tests
- Testers/Management underestimate the effort required to manage and maintain the test harness. The test harness should be treated as a software development project in its own right.
- Most Testers have never developed software, or have only done so for small projects. Developing software as part of a large development team takes practice.
- Testers usually aren't allowed to modify the application source code in order to add any hooks needed by the test harness.
- Testers knowledge of related development processes like SCM, and Design patterns may be limited. This often increases the complexity of the test harness, and decreases the overall quality and maintainability of the test harness.
All of the problems above can be addressed by simply shifting the test harness development responsibility, from the Testers, to the developers. Many developers may be resistant to this, but at the end of the day its their job to produce high quality, thoroughly tested code. The functional tests just take the level of testing performed by developers one level higher, by verifying the various components the developers have unit tested actually interact as expected.
It is important to note that we are using the automated functional tests to perform acceptance and regression tests of the system under test. It is not necessary to write a test for every single relationship between the components of the system. A compromise is needed, for example you could only automate the positive tests–tests that ensure the system behaves as expected when provided with valid input data.
The criteria for what is automated and what isn't automated will differ from project to project and is dependent on a number of other factors like:
- available tools
- development resources available
- project time-frames
- Management and Testers decide what functional tests need to be automated NOT developers.
- Developers need to take into account the time needed to write the automated tests when providing estimates for development tasks.
- Developers are responsible for providing Testers with a convenient way to run the tests.
- Developers are responsible for the structure and maintenance of the test harness.
- Testers are responsible for providing what ever information the developers need in order to complete a test and make it pass.
- Testers should provide a flow/state document for each automated test, detailing what the developer needs to check in order to validate a requirement.
- Testers are responsible for ensuring the tests written by developers are accurate, this could be in the form of a code review, or simply sitting with the developer as they run through the test having each part explained to them.
- A Single Developer and Tester need to be responsible for test harness. The developer has control over the structure of the test harness code, whilst the tester guides what tests are included in the test harness and how successes or failures are determined. This pair should also provide mentoring and guidance to the rest of the development and test teams to help the understand