if they find defects though. They are also good if you can run through things that the product is supposed to do, and they do not find anything. Now, they have not given you the same kind of information as if you find defects, but you want to be able to explore different areas of the product and test different areas of the product. I tend to work a lot on embedded kinds of systems, applications that are close to the operating system. I like to have a whole suite of regression tests. On the other hand, I have also worked on a lot of, a few I should say, really short time-to-market driven projects where we didn't necessarily want to run a lot of regression tests because we wanted to keep exploring the new development that was coming into the product on a daily basis. So, we had a lot of manual tests in the second case where we had a very short project. And we had a lot of automated tests for the longer projects where I wanted to know that we had not broken anything over a long period of time. So, if you aren't finding defects with your tests, it doesn't mean that your tests are bad. It means that you are not finding defects with your test and you might need to consider an alternative technique for finding tests, or I should say for finding defects. So, I think that is really important. It is also extremely difficult to measure how good a tester is by the number of defects or the kinds of defects that they report. Some testers tend to take a very systematic approach and they start from what they think of as the beginning, and they walk through all of the menus and they walk through the product in a way that makes sense to them, checking for things that the product is supposed to do. Eventually, they check for things that the product is not supposed to do, and that is a very systematic approach. It is an approach that I tend to take.
There are other people, though, who don't think the way I do and they do what seems to me incredibly random stuff. And then they find really good, or I should say, really bad defects. Those big bad defects that would prevent you from shipping or would be an incredible embarrassment to you while shipping.
Carol: Right.
Johanna: So, it's not that one kind of testing is better than another. Your best bet is to have a mix of people, some of whom sort of walk through the product systematically and some of whom are incredibly good exploratory testers. And that you want a mix of looking to see how much it is going to cost you, of course, of some form of automated regression tests along with newly introduced tests. I find that when I work on a project, especially as the test manager, I track the number of times requirements change over the course of the project. Now, the way I track them is sort of major and minor. It is a major thing if it is an essential requirement. It is a minor thing if it is only a desirable requirement. So, I try to look at how essential is this? Is this a constraining attribute of the product? Do we not have a product without this, or is this something that we are going to make the user have much more satisfaction if we include this thing? And I track major and
About the author
Johanna Rothman, known as the “Pragmatic Manager,” helps organizational leaders see problems and risks in their product development. She helps them recognize potential “gotchas,” seize opportunities, and remove impediments. Johanna was the Agile 2009 conference chair. She is the technical editor for Agile Connection and the author of these books:
- Manage Your Job Search
- Hiring Geeks That Fit
- Manage Your Project Portfolio: Increase Your Capacity and Finish More Projects
- The 2008 Jolt Productivity award-winning Manage It! Your Guide to Modern, Pragmatic Project Management
- Behind Closed Doors: Secrets of Great Management
- Hiring the Best Knowledge Workers, Techies & Nerds: The Secrets and Science of Hiring Technical People
Johanna is working on a book about agile program management. She writes columns for Stickyminds.com and projectmanagementcom and blogs on her website, jrothman.com, as well on createadaptablelife.com.
