A defect management system contains data such as how many defects have been raised, the priority and severity of individual defects, and even who is raising them. This information is regularly used by program and test management to guide decision making. In this article, Dan Minkin proves that an experienced test manager can gather useful information by looking at more than just the defect management system's data.
As a defect manager, or more specifically, as a test manager in charge of defects, I was never so popular as when I started to use a whiteboard to keep up-to-date with the latest found and fixed defects. Suddenly senior program management knew who I was and what I did. Every day they'd ask how many new defects were raised, what the priorities were, which ones were critical, and even who was raising them-data I could readily provide from my whiteboard. As the numbers went up, there would be more concern; as they came down, less. But as time passed, I realized that a lot of the useful information that was found on the defect management system couldn't be reported using metrics. These were the hidden messages that were important to decipher.
Defect Description and Developer Comments
Much of what turned out to be useful information was found in the defect description and consequent developer comments. What mattered was how it was written, not what the information was. While some testers would report defects in step-by-step detail, including preconditions and all relevant data, others might simply state that the system had crashed or that a calculation was wrong. Everybody, it seemed, developed his own style.
The large variety in styles used to report a defect told me a number of things. It told me who were the experienced and confident testers and who were not so confident. It told me which technical areas caused frustration, which became apparent in the language ("surely the developers can see..." "this is the third time..."), and, in so doing, added to the weight of evidence that certain functional areas were high risk.
Also, the developer's replies and consequent debate told me something about the relationship between the developers and testers. Again the language defined much of this, providing some clear evidence that relations weren't as good as they could have been. The number of replies-often three, four, or five from each side-indicated that either the defect resolution process wasn't working and that improved contact between development and the test team was needed, or that there was some confusion about what the requirements were. As it happened, both were true.
I had always been quite happy that the program management was committed to using a test management tool. Management had invested a sizeable amount of money in buying the tool and extra defect module licenses. But was this commitment being reflected by the testers and developers? I asked myself whether or not the defects were being kept up-to-date, whether descriptions and replies were too technical or conversely too business focused in nature, and did it seem like the suggested approach to defect description was being followed or was lip service being paid? Were all the fields correctly filled in or were the defaults just being taken? What became clear was that some people revelled in the use of the tool and used it to its fullest capabilities, and others who were used to different ways of working, needed coaching and encouragement to use the tool. This was hardly a groundbreaking revelation, but it was a useful one nonetheless. It subsequently led to the discovery that those who didn't relate to the tool often related worse to other elements of the formal testing process, such as test specifications.
The history of a defect sometimes gave me interesting information. Some defects had extensive history that detailed the status and ownership changes, some being open for several months. Once, when trying to track down why a given defect had been closed months before (with no explanation), it was found that on the day of closing, the defect had gone through a number of hands. Following through with the parties in question led to the discovery that a wrong assumption had closed this and other defects incorrectly. The defects were reinstated. On another occasion, we tracked another group of defects that had been allocated to an incorrect party. Again, studying the history enabled us to rectify incorrect details and feed lost defects back into the workflow.
Defect Fields and Formats
Prior to taking a holiday, I prepared a handover note for the defect management process. During the preparation, I realized how complex the defect management system was becoming. The number of different defect statuses should have given this away. There were seventeen. Seventeen was OK but was becoming unwieldy and difficult for other testers to comprehend. There were also twenty fields for testers to fill in. A common reason we cited for why testing took much longer than expected was because having to raise and detail so many defects was an onerous task. However, it didn't occur to me until this point that this was partly of our own making and not just a result of defect numbers.
I resolved to try and reduce this complexity by streamlining the process, eliminating a number of the mandatory fields, reducing the number of possible field values, and removing some of the statuses. This had a small but important effect on the speed of the defect process. Though we accepted that capturing the correct data up front was time consuming, it did mean less time was spent in the number of informal queries returning from developers.
I asked myself the following questions: Was the tool too complicated to use? Would someone else be able to follow the system intuitively or would it require training or copious documentation?
The answers to these questions told me something about the likely efficiency of the testing effort-that it wasn't as efficient as it could have been.
What new fields had been added? Why?
I'd been asked to add a field for retest failures. Was this because someone felt that the software was likely to need many patches? Or someone didn't trust that any given fix would work the first time? Was there a development lifecycle problem that needed to be addressed? Was it a problem with the ability of the developers? In the end it transpired that there was a political agenda, which was useful to know.
Which fields were being used? Did this indicate that the testers were focusing on the right things?
Filling in the business severity more accurately than the testing priority indicated that testers were focused on business rather than testing impact (not surprising since many were business second-ees). Therefore, their focus was somewhat too narrow for the role they were supposed to be playing. A second interesting example was the functional area field. Some testers always put the same value in this field rather than using it to describe the function in which a defect had been found. Discussions with these testers determined that they did not understand how the area that they were testing fit in to the overall test effort. Consequently, we spent some time educating these testers.
We discovered a useful piece of information when we asked why the test reference field was rarely being used. It turned out the majority of the defects were found using an exploratory approach rather than following scripts. This in turn indicated that there was a problem with the quality of the supplier testing. The supplier was using our test cases to test its software (in a rigid and incorrect fashion) and at no point experimenting or trying to break the system. This turned out to be one of the key discoveries of the project. As a result, two further intermediate stages of testing were inserted, first into the supplier test cycle and then into the user testing (a confidence test immediately after delivery).
Defect Tool and Its Use
As the program moved later into the testing cycles, the use of the tool became more widespread and higher profile. Who was using the tool, when, and why gave me insight into the program dynamics and events.
The program manager would ask me what the latest figures were, and explain to me why he needed to know. By knowing his reason, I was able to add extra information or pinpoint defects as the situation required.
By asking myself who was interested in the information it became possible to identify and build up relations with key players in the program. It became clear that one of the strand coordinators (who reported to the program manager) was really one of the driving forces of the program. The relationship that was fostered was mutually beneficial with the coordinator having readily available information on software quality and the test team having an active and vociferous ally.
The statistics that I provided on a daily basis were simple. My daily report explained how many new defects had been raised, the total number of defects to date, and how many defects existed with our supplier. No one ever asked me for historical data or how many defects had been fixed by our supplier since the start of the program or the average time it took to fix a high-priority bug.
It was just as well. I couldn't have answered any of those questions anyway. I hadn't set up the defect system to capture that kind of information. Once a defect was fixed, I wasn't interested in it anymore. That raised questions in my mind. Should I have planned for the time when the questions would be asked? Was this program simply not interested in carrying forward statistics into future programs? When I questioned this with the program test manager his view was firmly the latter. As a matter of pragmatism this issue dropped into the background. At least I knew two things: Program management didn't need me producing trends and lessons learned (despite my reservations, I accepted that it was too late to change this), and if I did it again, I'd spend more time analyzing the role of the defect management tool and the data it needed to capture rather than letting it develop over time.
The defect management tool and process should be a guide. Statistics and defects are useful, but they are only a window onto the health of the program. As ever, it is the awareness of the human element of any part of the program that makes for the full story. Elements to be aware of include the writing style, the amount of "debate" within the defect descriptions, and how successfully the tool is being used. A successful test manager should not take data and information merely at face value but should use this to inform his view and to ask himself further questions. So when you are looking at statistics from your defect management tool, know that there is more you can understand. The following questions can help you add more to what you already know:
- Do I detect emotion within the defect description?
- Is the tool being used as successfully as it could be? If not, why?
- Does the choice and use of defect fields shed any light on the testing process?
- Who is interested in the data that the tool produces?
- Do these reports tell the full story?
- Does the defect history shed light on the defect process?