A Small Matter of Metrics

You're thinking that now is a good time to start a software process measurement program in your organization. To prepare, you've read one or more books on measurement. You have a good idea about what data you'll need to collect and how you'll analyze it. But you've also heard stories about how similar efforts have failed in other organizations—stories that don't jibe with what you've read, suggesting that there may be more going on than what can be covered in a book.

What can you do to fill in the gaps? Consider running a trial measurement program on yourself before launching into a bigger program. By doing a practice program "in the small," and applying introspection and imagination, you can learn about the value and limits of measurement, gain insights into why many software measurement efforts fail, and prepare yourself for launching a successful program. You'll need only a spreadsheet that can produce graphs—Microsoft Excel or the equivalent is more than sufficient. A small stack of sticky notes is also useful.

Start Simple
On a new spreadsheet, label three columns: Date, Arrival, and Departure. For the next few weeks, record your arrival and departure times at work. (For the first few weeks at a new job, I also track commute times.) For bonus points, make a chart on a second spreadsheet page, and link it to the live data so that the chart updates when you add new data. This is a good excuse for experimenting with the graphing capabilities of your spreadsheet—a skill you’ll need later. Try out various chart types to see what they reveal about the data.

After a few weeks of recording times, you might notice a few things:

There will be missing data. Unless you're very well organized or leave yourself good reminders, you might forget to note or record an arrival or departure time. It's easy to forget or get caught up in a crisis. One lesson from this is that any measurements you gather, and any metrics you calculate, must be able to survive missing data.

What gets measured will get improved. You might observe that the mere act of recording your arrival time affects your motivation to get in to work on time, or stay a bit later (or leave at a saner hour). By measuring something, you focus your attention on it, albeit briefly. In a larger metrics program, focusing a team's attention on the right things, even if only for a few moments a day, can have a strong, positive effect on the team's behavior.

Numbers don't tell the whole story. From arrival and departure times, you can calculate and chart time on the job. Or can you? What about time you put in away from the office? You need to extend your data collection to account for non-office time. Can you figure out a way to do this that doesn't complicate data collection or charting? Getting precise measurements on some things can be complicated. The harder data collection becomes, the greater the temptation to avoid the complication and stay with simple, yet imprecise, measurements.

Charts are compelling. Trends are easier to spot when looking at a good chart than when reading raw numbers. Charts are also effective ways to present data to managers who have limited attention spans but still need to make informed decisions.

Add Other Measures
You might reasonably object, "But what does time on the job have to do with anything? I could sleep at my desk and the numbers would still look good. Shouldn't I be measuring how much effort I'm putting in, or how