My last column, Three Keys to Test Automation, stated that successful test automation requires team commitment, teamwork between testers and developers, and getting an early start. Jack Baseley replied with a very good question: "How do you propose to deal with late starts?" Do we just give up?

Jack asks:
My group just started an automated project and this article was a great help. My only concern is that an early start for automation is not always possible. How do you propose to deal with late starts?

My previous column identified two particular problems with late starts: you may not be able to get either the testability changes you need or the test strategy changes that you want. These circumstances make automation more difficult and may even prevent you from automating parts of the test suite.

Thus, I have a couple questions for those of you getting a late start. Why wasn't automation started earlier? And why do you want to do it now? Test automation is an investment.

In general, the later you start, the more you need to invest before you start seeing returns. Does the cost benefit analysis look better now that it did earlier?

I already said that commitment is key. With a late start it will be doubly so. You are going to need to double up on your staff. You'll still need to maintain your manual testing staff while the automation is developed. And you'll need a separate automation staff to get that effort off the ground without derailing your existing testing activities. Can your company make this kind of commitment to automation? And this is only the start.

An even bigger commitment will need to be made to changing your testing strategy. Testing strategy includes the way test cases are designed and described, how testing progress is reported, how tests are grouped and assigned, and how decisions are made regarding what tests need to be run when. All this changes with automation. And efficiency will require further changes. People resist change and often for good reason. Change entails risk. Manual testers may be concerned about their role once automation is implemented.

Managers may be uncomfortable having to learn new tradeoffs and having fewer eyes on the software. Changing test strategy on a mature product means taking avoidable risks.Because of the problems with change, I've seen test automation efforts that try to minimize the changes in test strategy and staffing. Invariably, however, this requires architecting the automation inefficiently. Lots of extra scaffolding is constructed to make the automation run the tests just like the manual testers. And they suffer from the fact that automation just can't be as discriminating as a pair of human eyeballs. So things still get missed.

The approach I now take is to try and follow the existing test strategy to the degree that it does not burden the test automation, but then to insist on the strategy changes that are required to allow the test automation to be reliable and efficient. These test strategy changes may make some managers uncomfortable. They may require changing how some tests are executed or how test results are verified. And to avoid being overwhelmed by false alarms, fewer things are checked.

I can understand why this may be more than some managers are prepared for. Sometimes test automation is seen as just a faster and easier way to get what you are already getting with your manual testing. That is not how test automation works. I avoid projects where I can't change these expectations.

My previous column also discussed the importance of testers and developers working closely together. With a late start this also will be even more critical. I have yet to work on a software project that didn't see significant technical problems in getting necessary testability hooks into the software. With a mature product, these testability hooks may need to be inserted into code that developers would otherwise not want to touch. Will your developers and managers be willing to potentially destabilize your code in the interests of assisting the automation project? If not, forget it.

Thus late starts with test automation entail changes to established testing procedures and changes to tested code.

Early in a project, changes in code or strategy are less risky, because we have less confidence in what we are changing from. But with time, change becomes more difficult. I'd estimate that the difficulty increases by a factor of three or four. And our test automation project now requires additional skills. Now we need a change agent, someone who is effective at shepherding an organization through necessary changes. It's a lot to ask.

Further Reading
Three Keys to Test Automation, Bret Pettichord, StickyMinds.com, December 2000