Estimating Test Metrics

Member Submitted

The goal of this paper is to discuss ways to create simple and Easy-to-use metrics to do better estimates. Estimates are important to test leads and test managers for a number of reasons. Here are some: 1) How many test cases do I need to write? 2) Can I use any of the existing test cases? 3) How many existing test cases do I need to modify? 4) How long will it take to execute the test cases? 5) How many resources do I need to write and execute the test cases?

There are no easy answers to any of these questions and we can not and should not make up numbers out of thin air (although I have seen some do just that). Primarily, there are two ways to do estimates.

Estimating the estimates: This approach of doing estimate is merely a speculation or taking a guess. It is not to say that these are wild guesses. On the contrary, this are rather informed guesses and the information comes from previous projects. Thus, estimates are done using the previous projects as a baseline. One can argue that this makes it a valid estimate. There is some truth to this argument and besides estimates are not meant to be concrete. But if the purpose of doing estimates is to come as close as possible to an absolute, then there is a chance that this approach will fall short. If estimated numbers are far a part from the actual numbers, then we are probably not doing a good job on our estimates and as a result we will be writing too few or too many test cases than we should. In order to take the guessing out of estimates, we do need to create some kind of documents which will help us to do a better estimate where estimated numbers will reflect the actual numbers (within an acceptable difference). A difference of up to 5 to 10 percent may be considered acceptable.

Documenting the estimates: The discussion of documenting the process of doing estimates will lead us to create some simple metrics which are easy to use. I have discussed one such metric in my previous article (A discussion on process, metrics, automation & outsourcing) some time ago. I needed to have some kind of measuring unit which would tell me the depth and breadth of my test coverage in detail. With that in mind I had created a metric which looks something like this (after some minor changes).

Test Coverage & Estimates
User stories Description Test Scenarios New t/c Existing t/c Modified t/c Total t/c In Lists & Spreadsheet the user selects one or two lists and selects quick graph option from menu. In L&S create and select,
- One list and do a quick graph
- Two lists and do a quick graph
- More than two lists and do a a quick graph
- Two adjacent lists and an empty column and do a quick graph
- One empty column and another list and do a quick graph
5 n/a n/a 5; Add “Quick Graph” as a menu item in the Data menu Quick Graph as a last item in Data menu n/a/ n/a/ 1 1 Add “Program Editor” as a menu item in the Insert menu Program Editor as a last item in Insert menu n/a n/a 1 1 “Line Number” edit field only accepts numeric - Rational numbers (ex: 11, 22, -11, -4/11 etc.)
- Irrational numbers (ex: square Root of 2)
- Decimal
- Alphanumeric starts with alpha
- Alphanumeric starts with numeric
- Alphabetic
- Special characters
- Spaces
- Numeric
- No entry (blank)
- Numeric with spaces
- Numeric, spaces and numeric
n/a 11
Total t/c     7 10 2 19

This metric serves two purposes. First, it describes all the possible test scenarios and thus put the focus on test coverage and second, the document also serves as a guide to do an estimate on how many test cases we need to write for these user stories.

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.