There's an old proverb about a frog in boiling water: If you drop a frog in a pot of boiling water, it will leap right out. However, if you plop the frog into a pot of room temperature water and gradually turn up the heat, the small changes will go unnoticed until it is too late.
QA engineers can suffer a similar fate. Take my team, for instance. We were working on a complex Web site. We had written a comprehensive test plan and thorough test cases. We had done a good deal of preliminary testing on the first release and believed we had addressed most of the major issues. We felt prepared for the next release. When the second release came in, the testing team brought on new testers to help finish the testing effort, just as planned. The issues that the new members found were shocking! The new issues were being reported against previously tested features and areas of the product. The new engineers were finding all sorts of bugs, from GUI problems to a lack of features.
Why all the new bugs? Well, they weren’t really new. We had simply contracted a bad case of what I call Adaptive Testing Syndrome (ATS). ATS happens when, for various reasons, test team members become blind to the idiosyncrasies of the software and even accept them as a normal part of the design, much as the frog fails to notice the gradually increasing heat of the water. However, when a different tester, or maybe just a different set of tests, comes in contact with the software, the bugs become painfully obvious.
How We Contracted and Cured ATS
How did my team get ATS and how did we keep from getting burned again? The good thing about ATS is that it's easy to find the causes and the cure. First, when we wrote the manual test cases, we left them vague on purpose to encourage the engineers to use their imagination and creativity when testing the features. Though this worked at first, the test engineers eventually formed a habit of how to test the feature and didn't stray from that path. Testing the feature the same way each time is, on the surface, a good thing. However, each feature had at least three different ways to be invoked, but was being tested using only each engineer's favorite method. This was easy to remedy. We split the test cases so that we had one for each of the different ways the feature could be invoked.
Another problem was that the engineers had been testing only one or two assigned areas/features since the beginning of the project. They had been staring at the same bugs for so long that they didn't see them anymore. Once again, there was a quick and easy remedy. We shifted everyone's responsibilities. Everyone had a different area/feature to work on. This also helped cure the previous problem. Since the engineers were not familiar with the new areas, they became more creative, curious, and investigative.
How to Diagnose ATS in Your Organization
If you think your project might have Adaptive Testing Syndrome, have some of the new members on the team retest features previously tested by someone else on the current build. It's an easy way to familiarize the new testers with the application and can give you some important information. Once this is complete, look at the results from the two different testers. Is the new tester finding many new bugs? You can expect some deviation because of differences in test case execution, application complexity, or varying skill levels. Major differences, though, can be indicative of ATS. In any case, it is still worth investigating the test cases that produced the different results.
Longer projects are particularly vulnerable to ATS. Test engineers become so familiar with the written test scripts for one area that they stop using them. The excuse is, "I don't need to use the test cases because I have run them N times and can recite them in my sleep.” When I hear this, I take the test scripts and sit behind the engineer and watch the testing, keeping track of what is tested. We then go over the differences between the test I witnessed and the test script. Without fail, the engineer will miss an area or feature; but will also usually test something not on the script as well. This brings us to another way in which long projects are vulnerable to ATS: when the test cases were written, they were complete, but since then, new features have been added or modified. No one has updated the test cases to reflect the application's current feature set. This can cause features to go untested or erroneous bugs to be written. Be on the lookout for ATS at all times, but particularly during long projects.
Ways to Cope
If your project has ATS, you'll have to roll up your sleeves and get the project back on schedule. You will probably want to make engineers responsible for features different from, and preferably not related to, what they were previously testing to help cure the adaptive issue. When you do, you will need to estimate the time it will take the new people to retest the current set of features. This should be relatively easy, since you have already tested them at least once with the old people. The project schedule will need to be updated to include the new testing effort. It is critical to understand the updated schedule in order to plan resources and method of attack.
If the retest effort will not fit within the current schedule, try creative solutions such as staggered shifts, weekends, overtime, or additional staff. Do your homework! Look carefully at the issue and brainstorm for as many ways as possible that quality assurance could help improve the situation. For example, is it possible to work alternative schedules to give you more workable hours in each day? Do you have morning people (6 AM to 3 PM) compared to late risers (11 AM to 7 PM)? Putting them both on the project allows you to get a fourteen-hour shift without working overtime. If all of these fail, you may have to alter the original schedule.
Prior to making drastic adjustments, remember that the members of your quality assurance team are your greatest assets. Demanding more hours from them can have negative consequences. While this may fix an individual project, you risk losing engineers if you repeatedly expose them to this. Try to remember that there will be more projects on the horizon and possibly other challenges to face.
Keeping the test cases updated and in a single location will cure many ills. It is taxing to make sure that the test cases cover every way that a feature can be accessed, but this will pay dividends in the long run. Make sure that the QA engineers are using the appropriate test cases. After all, you've spent the time writing and updating test cases, so put them to use! As a side note, testing without a script or exploratory testing is a valuable tool to be used in conjunction with scripted testing.
Another benefit of this technique is that your engineers get significant exposure to multiple projects and technologies. It also fosters a team mentality; everyone gets an opportunity to work with each other. You also avoid having engineers that are assigned to one project, not to be seen again for months.
In any testing effort, understand that the results may be flawed. If your engineers are unknowingly suffering from Adaptive Testing Syndrome, you may not catch these flaws until the product falls into your customer's hands. They have never seen the product before, at least in its current form, and will be sure to be ATS-free. Quality will suffer. Even if you have to change the project's schedule or budget, you may want to do it to avoid sending out a bad product. Making a wrong decision will not kill a project; but failing to change a bad decision may.