Estimating Test Metrics

Member Submitted

The goal of this paper is to discuss ways to create simple and Easy-to-use metrics to do better estimates. Estimates are important to test leads and test managers for a number of reasons. Here are some: 1) How many test cases do I need to write? 2) Can I use any of the existing test cases? 3) How many existing test cases do I need to modify? 4) How long will it take to execute the test cases? 5) How many resources do I need to write and execute the test cases?

There are no easy answers to any of these questions and we can not and should not make up numbers out of thin air (although I have seen some do just that). Primarily, there are two ways to do estimates.

Estimating the estimates: This approach of doing estimate is merely a speculation or taking a guess. It is not to say that these are wild guesses. On the contrary, this are rather informed guesses and the information comes from previous projects. Thus, estimates are done using the previous projects as a baseline. One can argue that this makes it a valid estimate. There is some truth to this argument and besides estimates are not meant to be concrete. But if the purpose of doing estimates is to come as close as possible to an absolute, then there is a chance that this approach will fall short. If estimated numbers are far a part from the actual numbers, then we are probably not doing a good job on our estimates and as a result we will be writing too few or too many test cases than we should. In order to take the guessing out of estimates, we do need to create some kind of documents which will help us to do a better estimate where estimated numbers will reflect the actual numbers (within an acceptable difference). A difference of up to 5 to 10 percent may be considered acceptable.

Documenting the estimates: The discussion of documenting the process of doing estimates will lead us to create some simple metrics which are easy to use. I have discussed one such metric in my previous article (A discussion on process, metrics, automation & outsourcing) some time ago. I needed to have some kind of measuring unit which would tell me the depth and breadth of my test coverage in detail. With that in mind I had created a metric which looks something like this (after some minor changes).

Test Coverage & Estimates
User storiesDescriptionTest ScenariosNew t/cExisting t/cModified t/cTotal t/c Lists & Spreadsheet the user selects one or two lists and selects quick graph option from menu.In L&S create and select,
- One list and do a quick graph
- Two lists and do a quick graph
- More than two lists and do a a quick graph
- Two adjacent lists and an empty column and do a quick graph
- One empty column and another list and do a quick graph
5n/an/a5;Add “Quick Graph” as a menu item in the Data menuQuick Graph as a last item in Data menun/a/n/a/11 “Program Editor” as a menu item in the Insert menuProgram Editor as a last item in Insert menun/an/a11“Line Number” edit field only accepts numeric- Rational numbers (ex: 11, 22, -11, -4/11 etc.)
- Irrational numbers (ex: square Root of 2)
- Decimal
- Alphanumeric starts with alpha
- Alphanumeric starts with numeric
- Alphabetic
- Special characters
- Spaces
- Numeric
- No entry (blank)
- Numeric with spaces
- Numeric, spaces and numeric
Total t/c  710219

This metric serves two purposes. First, it describes all the possible test scenarios and thus put the focus on test coverage and second, the document also serves as a guide to do an estimate on how many test cases we need to write for these user stories. It is always a good idea to have a quick review of this document to see if we are duplicating or missing a test scenario. It is also possible that we may want to combine two or more test scenarios into one test case and thus changing the number of test cases we need to write. For example, in case of the last user story (, we may want to have a Data Driven Test case (DDT) where all the test scenarios are listed in a spreadsheet and thus making it one test case instead of eleven. This metric is fairly easy to create, easy to follow and easy to maintain.

In addition, we may also want to use the following metric to identify the regression test cases associated withthese user stories. This metric is not driven by user stories or requirements as it is the case for the previous metric. Here the layout is completely different and focus is on different controls (Buttons, edit fields, combobox, spin control etc.). Below is the example of an estimate for the same user stories.

Checklist for Estimates
StandardCriteriaNew t/cExisting t/cModified t/c
FunctionalFunctional tests include all aspects of a dialog where there is underlying code that performs a certain task or executes a certain function.   
Dialog InitializationGo to Line dialogn/a1n/a
Close dialog (Esc, Click x)Go to Line dialogn/a2n/a
Default valuesDefault value for Line Number is always zeron/a1n/a
ButtonsOK and Cancel buttons in Go to Line dialog work correctlyn/a2n/a
Spin ControlLine Number only allows a max of current lines in Program Editor Line Number only goes down to zeron/a5
Combo boxLine Number only allows a max of current lines in Program Editor Line Number only goes down to zeron/an/an/a
Edit fieldLine Number only accepts numeric210n/a
Hot keyCtrl+Gn/a1n/a
GuiGui tests will test aspects of a dialog that are not covered under the functional tests. These include the placement of controls and labels in the dialog, properties of the controls, accelerator keys, etc.   
Minimize/MaximizeGo to Line dialogn/an/an/a
Tab orderGo to Line dialogn/a1n/a
Menu ItemsProgram Editor
Quick Graph
Go to Line
Tool tipGo to Line dialogn/an/an/a
CursorGo to Line dialogn/an/an/a
Other (functional & Gui)Functional and Gui not covered so far   
Quick GraphQuick graph from L&S5n/an/a
 Total new t/c7 
 Total existing t/c25 
 Total modified t/c2
 Grand total:34

Here the total number of estimated test cases is 34 and it includes the regression test cases. Since one of the user stories ( Line Number edit field only accepts numeric ) has to do with the changes made in "Go to Line" dialog, it is crucial that we run all other test cases (GUI and functional) associated with the dialog and this metric allows us to identify those test cases. The important thing to remember is that these metrics are dynamic, meaning changes can be made as needed. For example, if new controls are introduced (i.e. scroll bars, check boxes etc), we can add them to the list. Once these metrics are in place, doing an estimate comes to be a simple matter of plugging the numbers in the appropriate columns. With such ease of use of these metrics, there is no reason for the trajectory of doing an estimate should include any speculation.

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.