Satisfying our customers is an essential element to staying in business in this modern world of global competition. We must satisfy and even delight our customers with the value of our software products and services to gain their loyalty and repeat business. Customer satisfaction is therefore a primary goal of process improvement programs such as Total Quality Management (TQM) and the Software Engineering Institute's Capability Maturity ModelSM.
So how satisfied are our customers? One of the best ways to find out is to ask them, using Customer Satisfaction Surveys. These surveys can provide Management with the information they need to determine their customers' level of satisfaction with their software products and with the services associated with those products. Software engineers and other members of the technical staff can use the survey information to identify opportunities for ongoing process improvements-and to monitor the impact of those improvements.
Focusing on Key Customer Quality Requirements
When creating a Customer Satisfaction Survey, our first objective is to get the customer to complete the survey and return it. If the survey is short and easy to complete, we increase our chances of this happening. If the survey is long and detailed, the recipients are more likely to set it aside to complete later...only to have it disappear into the stacks of other papers on their desks. The first step in creating a Customer Satisfaction Survey is therefore to produce a concise and engaging review focusing on issues the customers care about: their key quality requirements.
When determining this list of quality requirements, it can be helpful to start by looking to the software quality literature and selecting those requirements that are relevant to your specific products or services. For example, in his book Practical Software Metrics for Project Management and Process Improvement, Bob Grady discusses the FURPS+ quality model used at Hewlett-Packard. The elements of the FURPS+ model include Functionality, Usability, Reliability, Performance, and Support. The plus (+) extends the acronym to include quality components that are specific to individual products. A second example is the ISO 9126 standard, Information Technology-Software Product Evaluation-Quality Characteristics and Guidelines for Their Use, which defines seven quality characteristics for software product evaluation:
In his book Measuring Customer Satisfaction, Bob Hayes gives us an example of a support survey, based on the quality requirements of Availability, Responsiveness, Timeliness, Completeness, and Professionalism.
These general lists from the literature can be tailored to match the quality requirements of a specific software product or service. For example, if your software product has extensive user interfaces and is sold internationally, the ability to easily change the product to meet the needs of languages other than English may be a key quality requirement. An excellent source of information to use when tailoring is the people who work to create the software product or who provide the software services. They can have unique insight into their job functions and how they relate to meeting the customer's requirements.
Another mechanism for determining the customer's quality requirements is the Critical Incident Approach described by Bob Hayes in Measuring Customer Satisfaction. In this approach, customers are interviewed and each interviewee is asked to describe 5-10 positive and 5-10 negative encounters they have had with the product or service in the past. The incidents are then used to generate categories of "satisfaction items" based on shared common words used in the incident description. For example, both positive and negative statements about how long they had to wait for help when they phoned the technical service support line would be grouped together into a "length of wait for service" item. These satisfaction items are then used to discover key customer quality requirements. For example, the "length of wait for service" item could be combined with the "ability to schedule a convenient field representative service call appointment" item and the "number of people transferred to" item, then summarized as the quality requirement called Availability of Service.
Creating the Questionnaire
After selecting the quality requirements that will be the focus of the survey, the next step is to create the survey questionnaire. The questionnaire should start with an introduction that briefly states the purpose of the survey and includes the instructions for completing the survey. Figure 1 illustrates an example of a Software Customer Satisfaction Survey.
The introduction is followed by the list of questions. This survey has two questions for each of the quality requirements of Functionality (questions 9 & 10), Usability (questions 7 & 8), Initial Reliability (questions 3 & 4), Long-term Reliability (questions 5 & 6), Technical Support (questions 11 & 12), Installation (questions 1 & 2), Documentation (questions 13 & 14), and Training (questions 15 & 16). This adds redundancy to the questionnaire, but it also adds a level of reliability to the survey results. Just as we would not try to determine a person's actual math aptitude by asking them a single math question, asking a single question about each quality requirement reduces the reliability of predicting the actual satisfaction level from the measured level. The questionnaire also has two questions that judge the customer's overall satisfaction-one for the software product and one for the support services.
The questionnaire in Figure 1 uses a ratio scale of 1 to 5 to measure the customer satisfaction level. We could have simply asked the question "Are you satisfied or dissatisfied?" However, from a statistical perspective, scales with only two response options are less reliable than scales with five response options. Studies have shown that that reliability seems to level off after five scale points-so further refinement of the scale does not add much additional value. Note that this example also asks the customer to rank the importance of each item. We will discuss the use of the "importance index" later in this article.
In addition to the basic questions on the questionnaire, additional demographic information should be gathered to aid in the detailed analysis of survey results (not shown in Figure 1). Again, the questions included in the demographic section should be tailored for individual organizations. For example, the demographic information on a survey for a provider of large software systems that are used by multiple individuals within a customer's organization might include:
Finally, some of the most valuable information that comes from a Customer Satisfaction Survey may come to us not from the questions themselves but from the "comments" section. I recommend that each question include a "comments" section. This gives the respondents the opportunity to write down their comments as they are thinking about that specific question. Figure 1 demonstrates the placement of the comment areas, but on a real questionnaire more space would be provided for actual comments. I have found the following benefits from having a comment section for each question:
The last step in creating the questionnaire is to test it by conducting a pilot survey with a small group of customers. The pilot should test the questionnaire at the question level, insuring that A) each question produces a reasonable distribution of answers, B) the meaning the customer places on each question matches the intended meaning, and C) each question is not ambiguous, overly restrictive, or overly general. The pilot should also take a macro view of testing the questionnaire-looking for problems with flow and sequence of the questions, question order or grouping that induces a bias, and issues with the time, length, and format of the questionnaire. Terry Vavra's book Improving Your Measurement of Customer Satisfaction provides information on testing for and avoiding these mistakes.
Deciding Who to Ask
The responses to our surveys may be very dependent on the role the respondent has in relationship to our software product. For example, if we again look at large software systems for large multi-user companies, the list of individual customer stakeholders might include:
If, however, we are looking at shrink-wrapped software we might be more interested in customer groups defined by personal versus business use, or by whether they are using the product on a standalone PC or on a network.
In their book Exploring Requirements, Donald Gause and Gerald Weinberg outline a set of steps for determining user constituencies. The first step is to brainstorm a list of all possible users. This includes any group that is affected by or affects the product. The second step is to reduce this list by classifying them as friendly, unfriendly (users trying to obtain unauthorized information from the system), or users to ignore. For the purposes of Customer Satisfaction Surveys, we typically want to focus our efforts on those groups classified as friendly.
There are several ways of dealing with this diversity in Customer Satisfaction Surveys. First, you may want to sample from your entire customer population and simply ask demographic questions like those above to help sort and analyze the responses by customer group. Secondly, you may want to limit your survey to only one customer group. For example, if you notice that your sales have fallen in a particular market, you may want to survey only customers in that market.
When sending questionnaires to a sample set of customers, your goal is to generalize the responses into information about the entire population of customers. In order to do this, you need to use random sampling techniques when selecting the sample. If you need to ensure that you have adequate coverage of all customer groups, you may want to use more sophisticated selection techniques like stratified sampling. Both Bob Hayes' and Terry Vavra's books discuss sampling techniques.
Designing a Customer Satisfaction Database
Conducting any comprehensive Customer Satisfaction Survey will result in the accumulation of large amounts of data. Every item on the survey will have two responses (i.e., satisfaction level and importance) and-potentially-a verbose response to the "comment" area. Multiply that by the number of questions and by the number of respondents, add the demographic data…and the volume of data can become huge. I highly recommend that you create an automated database to record and manipulate this data. A well-designed database will also allow for easy data analysis and reporting of summarized information. The following paragraphs describe the basic record structure in an example relational customer satisfaction database. (For smaller, simple surveys, this database could be implemented using a spreadsheet. However, for large amounts of data I recommend that a database tool or statistical package be used to manage this information.)
Figure 2 illustrates the relationships between the records in a sample database. There would be one Customer Record for each of the company's major customers (e.g., a supplier of telecommunications equipment might have major customers such as Sprint, Bell South, and GTE). A Survey Record is created for each Customer Satisfaction Survey returned or interview completed. Multiple individuals at each company could complete one or more surveys, so there is a one-to-many relationship between a Customer Record and Survey Records. There is a Response Record for each question asked on the survey, creating a one-to-many relationship between the Survey and Response Records, as well as a one-to-one relationship between each Response Record and Question Record. This design allows different questions to be asked of different survey participants (e.g., different sets of questions for installers and users) and the flexibility of modifying the questions over time without redesigning the database. The Response Record also has a one-to-one relationship with a Comment Record if text was entered in the comment portion of the questionnaire.
Reporting Survey Results: Turning Data into Information
Customer Satisfaction Surveys generate a tremendous amount of data-but what does it tell you? The trick is to convert this data into useful information that can be utilized by managers and members of the technical staff. I have found two reports to be very useful: a summary of the current satisfaction/importance level for each question (allowing the user to quickly identify which areas are candidates for improvement efforts) and a report of satisfaction level trends for each area over time.
Current Satisfaction Levels
Figure 3 illustrates an example of a metric report that summarizes the survey results and indicates the current customer satisfaction level. To produce this report, the survey responses to all questions related to that quality requirement are averaged for satisfaction level and for importance. For each requirement, these averaged values are plotted as a numbered bubble on an x-y graph. These numbers correspond to the requirement number in the left column. The green area on the right of this graph indicates the long-term goal: an average satisfaction score of better than 4 for all quality requirements. The yellow area indicates a shorter-term goal of having an average score better than 3. Red bubbles indicate scores that meet the long-term goal, green bubbles indicate scores that meet the short-term goal, and dark blue bubbles indicate scores outside the goal.
Note that the long- and short-term goals do not take importance into consideration. Our goal is to increase customer satisfaction, not to increase the importance of any quality requirement in the eyes of the customer. If the importance level of a requirement is low, then one of two things may be true. First, we may have misjudged the importance level of that requirement to the customer, and it may not be a key quality requirement. (In this case we may want to consider dropping it from our future surveys.) On the other hand, the requirement could be important but just not high on the customer's priorities at this time. So how do we tell the difference? We do this by running a correlation analysis between the overall satisfaction score and the corresponding individual scores for that requirement. This will validate the importance of the quality dimension in predicting the overall customer satisfaction. (Bob Hayes' book Measuring Customer Satisfaction discusses this correlation analysis in greater detail.)
From the report in Figure 3, it is possible to quickly identify Initial Software Reliability (bubble 2) and Documentation (bubble 7) as primary opportunities to improve customer satisfaction. By polling importance as well as satisfaction level in our survey, we can see that even though Documentation has a poorer satisfaction level, Initial Software Reliability is much more important to our customers and, therefore, would probably be given a higher priority.
Figure 4 illustrates an example of a metric report that shows the distribution of satisfaction scores for three questions. Graphs where the scores are tightly clustered around the mean (question A) indicate a high level of agreement among the customers on their satisfaction level. Distributions that are widely spread (question B) and particularly bi-modal distributions (question C) are candidates for further detail analysis. When analyzing the current satisfaction level, the reports in Figures 3 and 4 can be produced for various sets of demographic data. For example, versions of these graphs could be produced for each customer, each software release, or each respondent role. These results could then be compared with each other and with overall satisfaction levels to determine if the demographics had any impact on the results. For example, is there a particular customer who is unhappy with our technical support? Has the most recent release of the software increased customer satisfaction with our software's functionality? Or are the installers dissatisfied with our documentation while the users are happy with it? This type of analysis can require detailed detective work and creativity in deciding what combinations to examine. However, having an automated database and reporting mechanism makes it easy to perform the various extractions needed for this type of investigation.
The "comment" data can also be very valuable when analyzing the survey data for root cause reasons for customer dissatisfaction. This is especially true if there are reoccurring themes in the comments-if one particular feature is mentioned repeatedly in the Functionality comments, for example, or if multiple customers mention the unavailability of technical support on weekends.
Satisfaction Level Trends over Time
Another way to summarize the results of our satisfaction surveys is to look at trends over time. Figure 5 illustrates an example of a metric report that shows trends in "Features Promised vs. Delivered" based on quarterly surveys conducted over a period of eighteen months. Again, the green and yellow areas on this graph indicate the long- and short-term satisfaction level goals. This particular graph has a single line indicating the overall satisfaction level for the product. However, when analyzing these trends, the demographic data can be used to create multi-line graphs where each line represents a separate classification (e.g., a customer, a software release, or a respondent role). This allows for easy comparisons to determine if the demographics had any impact on the trends.
One note of caution is that to show trends over time the survey instrument must remain unchanged in the area being trended. Any rewording of the question can have major impacts on results; and historic responses collected before the change should not be used in the trend.
The primary purpose of trend analysis is to determine if the improvements we have made to our products, services, or processes have had an impact on the satisfaction level of our customers. It should be remembered, however, that satisfaction is a trailing indicator. Customers have long memories-the dismal initial quality of a software version three releases back may still impact their perception of our product even if the last two versions have been superior. We should not get discouraged if we do not see the dramatic jumps in satisfaction we might expect with dramatic jumps in quality.
It's Better to Know
Customer Satisfaction is a subjective measure. It is a measure of perception, not reality, although when it comes to a happy customer, perception is more important than reality. One phenomenon that I have noticed is that as our products, services, and processes have improved, the expectations of our customers have increased. They continue to demand bigger, better, faster. This can result in a flat trend on our survey graphs, even if we are continuously improving…or worse still, a declining graph because we are not keeping up with the increases in our customers' expectations. But take heart; even this is valuable information that we need to know in the very competitive world of software.
Labeled Measurement Scales: A Common Mistake Made In Surveys
If you want to achieve results that can be compared mathematically with other survey findings, avoid fully-labeled measurement scales such as:
1 Very Dissatisfied
5 Very Satisfied
There is not an exact distance between each value in the scale. In this example, the magnitude of the difference between "very dissatisfied" and "dissatisfied" is not the same as between "dissatisfied" and "neutral." In fact, these intervals can vary greatly with the perceptions of each person answering the survey.
Without an assumption of magnitude, the mathematical operations of addition, subtraction, multiplication, and division cannot be used on measurement values. This means that taking an average and saying that the average satisfaction level is 4.2 is meaningless. To avoid this problem, label only the endpoints of the scales, as is done in the sample survey in the article.
A Sample Software Customer Satisfaction Survey
The ABC Software Company is committed to quality and customer satisfaction.
We would like to know your opinion of our XYZ software product. Please indicate your level of satisfaction with each of the following characteristics of our product, as well as each item's importance to you.
On a scale of to 5, circle the appropriate number that indicates how satisfied you are with each of the following items-a score of 1 being very dissatisfied (VD) and 5 being very satisfied (VS).
On a scale of to 5, circle the appropriate number that indicates the importance to you of each of the following items-a score of 1 being very unimportant (VU) and 5 being very important (VI). Note that items 17 18 do not have importance scores since they are overall satisfaction items.
In the Comments section after each question, please include reasons for your satisfaction or dissatisfaction with this item, including specific examples where possible.