Documentation Metrics

What Do You Really Want to Measure?

"It is inevitable that sooner or later, someone higher up the food chain is going to ask you for documentation metrics." In this article, Donald Le Vie distinguishes between "mechanical metrics" that don't really measure your productivity, and "quality metrics," which are more difficult to use, but much more accurate. Le Vie provides excellent examples of these, and analyzes the value of various types of metrics.

This article reprinted with permission from INTERCOM, the magazine of the Society for Technical Communication (STC).

Regardless of the industry in which you work, it is inevitable that sooner or later, someone higher up on the food chain is going to ask you for documentation metrics. Metrics can be defined as a system for measuring production or product standards, so naturally somebody, somewhere will want to know how "productive" the technical writers and editors can be. Or they will want to measure the quality of your work. Much of the time, the types of metrics used seem to have little correlation with the value technical communicators actually provide.

Because these measurements can be critical to your success in an organization, there is a danger in relying on standards that offer little in real value. Rather than continuing to go along with the game, technical communication professionals should exert some influence on the types of measurements applied to documentation projects.

This article focuses on metrics you should avoid—hopefully, this will give you the ammunition you need to defend yourself against managers and bean counters who want to impose arbitrary standards on your work. This article also details metrics you should embrace so that you can get moving in the right direction.

Metrics: An Overview
The theory of metrics is that some properties can be measured and that some relationship exists between what we can measure and what we want to know. At times, it can admittedly be difficult to relate what we can measure to some quality attribute(s). You can easily count apostrophes per page, for instance, but that doesn't tell you anything about the quality of the writing.

I've been involved with defining, collecting, and interpreting metrics for more than twenty years. I've seen metrics adapted for all sorts of different reasons. When I worked as a research geological oceanographer for the National Oceanic and Atmospheric Administration in 1970s, for example, choosing the proper analytic standards for research was critical to obtaining continued funding. The metrics used in the field and in the labs were great for studying sediment dispersal patterns on the continental shelf and the engineering stability of seafloor muds. But the U.S. Department of Commerce was ultimately interested in only one thing: What kind of bang were the taxpayers getting for the buck? Were the results of our efforts meaningful to the people paying the bills? We had to translate the results of our field and lab evaluations into a language taxpayers could understand: dollars, as expressed in risk assessment costs, environmental assessment costs, and contingency costs associated with the economic development of offshore areas. These became our metrics, and they measured our successes.

Starting Right
It is important to define standards for both product and process right from the start. The type of measurement to be used and the questions to be answered with respect to quality should be determined at the outset of the project. The fundamental question of "What do we really want to measure?" should be addressed in the documentation project plan, before any writing is done. From the very beginning, raise the question, "How does this metric contribute to a qualitative assessment of our information products?" If you start by asking, "What’s easy to measure?" you'll probably end up with a meaningless metric. You might as well ask, "How many angels are dancing on the head of this pin?"

Another way to put it is that you should strive to use quality metric rather than mechanical metrics . A quality metric is a metric intended to be a predictor of document quality (though the relationship between these metrics and quality has to be examined carefully). When we assign


About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.