product metrics

Conference Presentations

The Dangers of the Requirements Coverage Metric

When testing a system, one question that always arises is, “How much of the system have we tested?” Coverage is defined as the ratio of “what has been tested” to “what there is to test.” One of the basic coverage metrics is requirements coverage-measuring the percentage of the requirements that have been tested. Unfortunately, the requirements coverage metric comes with some serious difficulties: Requirements are difficult to count; they are ideas, not physical things, and come in different formats, sizes, and quality levels. In addition, making a complete count of “what there is to test” is impossible in today’s hyper-complex systems. The imprecision of this metric makes it unreliable or even undefined and unusable. What is a test manager to do?

Lee Copeland, Software Quality Engineering
The Metrics Minefield

In many organizations, management demands measurements to help assess the quality of software products and projects. Are those measurements backed by solid metrics? How do we make sure that our metrics are reliably measuring what they're supposed to? What skills do we need to do this job well? Measurement is the art and science of making reliable and significant observations. Michael Bolton describes some common problems and risks with software measurement, and what we can do to address them. Learn to think critically about numbers, what they appear to measure and how they can be distorted. Improve the quality of the information that we're gathering to understand the relationship between observation, measurement, and metrics. Evaluate your measurements by asking probing questions about their validity.

Michael Bolton, DevelopSense, Inc.
Simple Metrics for Starters

Measurements and metrics are a hot topic again. Theorists often rail against them as meaningless and potentially harmful. Practitioners fear them because they don’t want to be metric-ed out of a job. Managers want them so they can better understand what is happening in the software development lifecycle and try to make their processes more efficient. David Gilbert shares the simple set of measurements and metrics Raymond James has implemented and describes the practical benefits they have gained. By first asking stakeholders what they really wanted to understand and then developing metrics to support their goals, David and his team have made measurement work at their company. They educated the stakeholders on key metrics concepts-first order measures and subjective relativity-and established a common model everyone agreed to follow.

David Gilbert, Raymond James
Talking Quality to Business: Metrics for Improvement

Are your testing and quality assurance activities adding significant value in the eyes of your stakeholders? Do you have difficulty convincing decision-makers that they need to invest more in improving quality? Selecting metrics that stakeholders understand will get your improvement project or program past the pilot phase and reduce the risk of having it stopped in its tracks. Todd Brasel and Kent McDonald show you how to avoid getting bogged down in the minute details of test results reporting and approach quality measurement from a systems perspective. They offer practical lessons about using metrics to promote quality improvement initiatives to upper managers and executives-the people who make the real decisions about quality. Learn about common traps and problems of measurement programs and how to avoid them.

Todd Brasel, Pitney Bowes
A New Paradigm for Collecting and Interpreting Bug Metrics

Many software test organizations count bugs; however, most do not derive much value from the practice, and other metrics can actually harm the quality of their software or their organization. Although valuable insights can be gained from examining find and fix rates or by graphing open bugs over time, you can be more easily fooled than informed by such metrics. Metrics used for control instead of inquiry tend to promote dysfunctional behavior whenever people know they are being measured. In this session, James Bach examines the subtleties of bug metrics analysis and shows examples of both helpful and misleading metrics from actual projects. Instead of the well-known Goal/Question/Metric paradigm, James presents a less intrusive approach to measurement that he describes as the Observe/Inquire/Model. Learn about the dynamics and dangers of measurement and a new approach to improve your metrics and the software you produce.

James Bach, Satisfice Inc
A "Follow the Sun" Test Automation Strategy

In this case study of an award winning project, Andy Redwood describes how his team used "best shoring" of testing services to reduce costs, reuse assets, and get the best from their test automation tools. In an enterprise-wide transformation process at a large investment bank, his team used available infrastructure, technology, tools, and process to reduce business risk from software changes with a new automated regression test suite. With some facts and figures and a little hindsight, you will learn how to provide global, automated testing services on a twenty-four hours a day/seven days a week, on-demand basis. Find out what metrics you need to accurately measure the costs and benefits of a "follow the sun" test automation strategy.

  • A successful outsource project that measurably improved business resilience
  • The do's and don'ts of offshore testing
Andy Redwood, Buttonwood Tree Group
Implementing and Sustaining a Measurement Program

Are you looking to install new measurements at the department or enterprise level? Are parts of your existing measurement program shaky? Starting a measurement program or revitalizing an existing one requires a good road map and checkpoints along the way. Janet Russac offers the fundamentals for establishing an organization-wide measurement program based on defined objectives. Find out about the principles of when to use metrics and when not to use them. Get a proven measurement program implementation strategy from this industry veteran, and take away an understanding of the key steps and attributes of a successful program. Make your measurements even more valuable by incorporating a benchmarking component into your program.

  • Key steps to a successful measurement program
  • Identification of key indicators of readiness and factors for success
Janet Russac, The David Consulting Group
Integrating Requirements-Based Tesing in the Development Process

Good data feedback of software measurements is critical when analyzing measurement data, for drawing conclusions, and as the basis for taking action. Feedback to those involved in the activities being measured helps validate the data as well. In this presentation Ben Linders shows examples of how Ericsson Telecommunications delivers feedback at two levels: projects and the total development center. Although the basics are similar, the application differs, and the key success factors depend on the level and the audience. At the project level, you will see how the team reviews defect data, including defect classifications and test matrices. For development center feedback, you will see how line management and technical engineers review data and analyze information based on a balanced score card approach with measurable goals.

Richard Bender, Technology Builders, Inc.
Techniques for Measuring Software Quality

A software defect is any flaw or imperfection in a software work product. A software defect is the result of a mistake in the design of a software process. In this article you will learn and gain an understanding of the software measurement processes, practices, and methods for: managing the quality of software products, improving the quality of software processes and procedures, and applying quantitative principles to your work for measuring quality.

James Rozum, Union Switch & Signal Inc
12 Steps to a Successful Metrics Program

Software metrics are an integral part of the state-of-the-practice in software engineering. More and more customers are specifying quality metrics reporting as part of their contractual requirements. Companies are using metrics to better understand, track, control, and predict software projects and products. This paper explains the basic concepts of metrics and measurement theory and how those
concepts relate to software.

Linda Westfall, The Westfall Team

Pages

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!