My company’s website, PNM.com, was getting a total redesign, which included adding many new features and self-service functions. The site was designed as a blend of adaptive and responsive design to provide the best possible experience for customers using PCs, tablets, and mobile devices.

We had written and executed more than a thousand test cases, but they were performed by professional testers using desktop PCs and a limited pool of mobile devices. We needed to get feedback about field use, where customers might use any device. As always, time was short.

Old website

New website

Crowdsourced testing is the process where testers test on their own terms, using their own devices and on their own time. This testing is used by all major web-based companies. Our solution was to do crowdsourced testing but to use our employees as the testers. The goal was to find bugs and risks to product adoption, not complete defined test scripts. This turned out to be a very effective way to get much larger test coverage of the universe of possible customer experiences. There were many benefits we experienced in using this testing method:

• As the local utility, most of our employees are also customers and had accounts set up in the test database, which was a copy from production.
• Our employees understand the business model of the company, so they could move forward with very brief, limited training.
• Employees could use their own personal devices and home networks located in our service area.
• Our employees understand what is important to our customers and could focus on those things.
• The testing process allowed our employees to get familiar with the new website so they could promote it to their friends and neighbors in conjunction with our marketing efforts.

We called our project the PNM.com Bug Hunt and marketed it to the employees in several ways. Test leadership sent postcards to our target audience in customer service and BTS, the internal technical group. These groups have a high level of technical skills and also direct experience related to what our customers will want most. We also placed posters throughout the company to attract a large and diverse testing pool and wrote articles for the company’s internal website.

Traditional crowdsourced testing is driven by paying for bugs found. One of our goals was also to get a large pool of participating testers, so we turned the exercise into a game, a process called “gamification,” and provided rewards to players, including branded prizes and cash.

Our testers reported bugs on a SharePoint page; they could also visit the page to see what the other hunters were reporting. By keeping the SharePoint site easy to use and available to everyone, we increased the chances people would make the effort to repot bugs, even at home or on mobile devices.

So, What Happened?

We found so many bugs in the first cycle that we ran a second cycle in order to build confidence for our site team—plus, our bug hunters were interested in doing more testing. We had:

• 56 bug hunters (the majority of user-acceptance testing cycles here have ten users at most)
• 561 bugs found and reported
• More than two hundred hours of testing time reported
• More than a hundred different devices used for testing (PCs, laptops, cellphones, tablets, etc.)
• 80 percent of the bug hunters reported that they tested to help the company
• 93 percent said they would do this testing again if asked

The cost of the crowdsourced bug hunt was less than $3,000, which included $1,300 in cash rewards.

We also were able to build an excited and experienced employee user group on the website prior to its going live publicly.

Conclusion

The Bug Hunt worked well for us in a number of ways. First, our testers were from all over the state, just like our customers. They also used many different personal devices, and we gained real confidence our site would not have major device or network issues when it went live.

The testers reported many bugs we were able to fix before the public rollout, which gave us more capacity to deal with the ones that slipped by after. And our customer support team was ready on day one for calls and questions about the new website because they had had time to get used to it.

The bug hunt was a cost-effective method to test our new website, and it had the extra benefit of educating our employees on the new features and functions of the site. And most importantly, our employees were involved and excited about the project and felt like an important part of the team.