What is crowdsourced testing and why is it useful?
The "crowd" is a diverse set of people from the community at large, who can be leveraged in testing and providing feedback on a product because of their end-user representation, domain knowledge, subject matter expertise, or software testing experience.
Crowdsourced testing is the process of using the crowd and employing a strategy to have them provide formal and informal feedback on the product under test. In my humble opinion, the most value from crowdsourced testing is reaped in the first three scenarios above, rather than relying on the crowd's testing experience alone, which you can largely get from your traditional testing methods. In the first three groups, the crowd tester is a more realistic tester than a trained tester.
The crowd can be engaged for monetary gains for the work they do, but this is not always the case. Pride of participation, non-monetary rewards, other accolades, sense of satisfaction, and to quench one’s personal or professional aspirations could be other drivers that engage the crowd.
It is often misinterpreted that crowdsourced testing is about engaging and working with a company whose business model is to pool a crowd of testers who get paid for valid bugs they report. Well, this is definitely one form of crowdsourced testing, but there are several other more important and creative manifestations to look at, to leverage the crowd's wisdom including:
- Sourcing relevant people from within one's company, although they might not be directly working on the particular project
- Building a pool of end-users gradually over time to provide feedback across various phases of product development (be it public beta users, dogfood programs, private betas covering Most Valuable Players, etc.)
- Partnering or working with organizations and universities who have domain knowledge and subject matter
What are some of the risks of crowdsourced testing?
Crowdsourced testing is not a no-brainer solution that is free of risks and challenges. Risks exist in two categories: implementation and psychological. On the implementation side, unless you make an educated decision of what, when, and how to crowdsource, the effort can soon become chaotic, overwhelming, and randomizing for both the crowd and the team that is managing the effort. Fortunately, there are clear mitigation strategies to alleviate the risks, in the form of best practices to adopt.
One of the biggest risks is that the crowd is not contractually bound to you or your product, unless you work with a very private crowd where you sign a non-disclosure agreement (NDA) with them. In cases where you do not have an NDA, you want to be selective about what you want to crowdsource to be able to leverage the crowd's inputs but not compromise your intellectual property.
Also, it is important to understand under what circumstances the crowd typically succeeds and try to maximize on them, so as to minimize risks. In my experience, I've seen crowdsourcing succeed when: 1) A diverse crowd is chosen to test the product, 2) independence is maintained in their communication unless the product requires them to talk amongst themselves to promote knowledge sharing, 3) the product has a global user base requiring vast domain knowledge that is difficult to source and live user environments that are difficult to simulate internally, and 4) when tasks assigned to the crowd align with factors that motivate them.
On the psychological side, unless adequately addressed, the core test team (be it your internal team or a vendor or outsourced test team) may feel threatened by the presence of a crowd team. This not only demotivates the core team but also reduces the chances of success for the crowd, due to lack of the core team's support for the crowd test effort. The sponsor needs to ensure that everyone involved understands that the crowd is really a supplemental team to your core team—not a stand-alone team—with each having its core competencies. When this is established, a sense of empowerment flows in, where the core team feels more confident and will go out of its way to support the crowd test effort.