What is crowdsourced testing and why is it useful?

 

The "crowd" is a diverse set of people from the community at large, who can be leveraged in testing and providing feedback on a product because of their end-user representation, domain knowledge, subject matter expertise, or software testing experience.

 

Crowdsourced testing is the process of using the crowd and employing a strategy to have them provide formal and informal feedback on the product under test. In my humble opinion, the most value from crowdsourced testing is reaped in the first three scenarios above, rather than relying on the crowd's testing experience alone, which you can largely get from your traditional testing methods. In the first three groups, the crowd tester is a more realistic tester than a trained tester.

The crowd can be engaged for monetary gains for the work they do, but this is not always the case. Pride of participation, non-monetary rewards, other accolades, sense of satisfaction, and to quench one’s personal or professional aspirations could be other drivers that engage the crowd.

It is often misinterpreted that crowdsourced testing is about engaging and working with a company whose business model is to pool a crowd of testers who get paid for valid bugs they report. Well, this is definitely one form of crowdsourced testing, but there are several other more important and creative manifestations to look at, to leverage the crowd's wisdom including:

• Sourcing relevant people from within one's company, although they might not be directly working on the particular project
• Building a pool of end-users gradually over time to provide feedback across various phases of product development (be it public beta users, dogfood programs, private betas covering Most Valuable Players, etc.)
• Partnering or working with organizations and universities who have domain knowledge and subject matter

What are some of the risks of crowdsourced testing?

Crowdsourced testing is not a no-brainer solution that is free of risks and challenges. Risks exist in two categories: implementation and psychological. On the implementation side, unless you make an educated decision of what, when, and how to crowdsource, the effort can soon become chaotic, overwhelming, and randomizing for both the crowd and the team that is managing the effort. Fortunately, there are clear mitigation strategies to alleviate the risks, in the form of best practices to adopt. 

One of the biggest risks is that the crowd is not contractually bound to you or your product, unless you work with a very private crowd where you sign a non-disclosure agreement (NDA) with them. In cases where you do not have an NDA, you want to be selective about what you want to crowdsource to be able to leverage the crowd's inputs but not compromise your intellectual property.

Also, it is important to understand under what circumstances the crowd typically succeeds and try to maximize on them, so as to minimize risks. In my experience, I've seen crowdsourcing succeed when: 1) A diverse crowd is chosen to test the product, 2) independence is maintained in their communication unless the product requires them to talk amongst themselves to promote knowledge sharing, 3) the product has a global user base requiring vast domain knowledge that is difficult to source and live user environments that are difficult to simulate internally, and 4) when tasks assigned to the crowd align with factors that motivate them.

On the psychological side, unless adequately addressed, the core test team (be it your internal team or a vendor or outsourced test team) may feel threatened by the presence of a crowd team. This not only demotivates the core team but also reduces the chances of success for the crowd, due to lack of the core team's support for the crowd test effort. The sponsor needs to ensure that everyone involved understands that the crowd is really a supplemental team to your core team—not a stand-alone team—with each having its core competencies. When this is established, a sense of empowerment flows in, where the core team feels more confident and will go out of its way to support the crowd test effort.

Under what circumstances is crowdsourced testing a better option than traditional testing?

To some extent, I've answered this question in my previous response on risks in crowdsourced testing. Typically, crowdsourced testing is a better option when the product has a global user base requiring vast domain knowledge that is difficult and ineffective to source full time and when it is difficult to simulate all of the end-user scenarios in house. The following are a couple of cases when the crowd is a very good resource to leverage:

 

• Testing end-user-facing features where the crowdsourced team represents the end-user mindset and behaviors
• Performance, localization, and compatibility testing that need to cover several end-user environments

Is crowdsourced testing appropriate for all projects?

No. Crowdsourced testing is not a straightforward solution for all projects. Intrinsic characteristics of the crowd, the product under test, and the project’s constraints play key roles in deciding whether or not crowdsourcing is going to be useful in a given case. Unless this is planned for effectively, bringing in the crowd adds a significant risk factor to the project, as discussed earlier. The following are some cases where you would be better off working with your internal core test team with atraditional testing model:

 

• Areas where you have a lot of moving pieces and dynamics that require a lot of internal collaboration
• Areas of very sensitive intellectual property
• Areas that are dependent on some internal hardware or software resources that are difficult to simulate externally, at least at the given point of time
• Test automation, test-driven development scripts, regression, integration testing, first levels of performance, security testing that require discussions with business teams to baseline metrics

How does one get started implementing crowdsourced testing in his organization?

Crowdsourced testing is an educated decision that needs to be made keeping in mind what, when and how to crowdsource. In my experience, the “"hen" factor is the most important, followed by "what," and finally "how."

The "When" Factor: It makes the most sense to crowdsource when your product is working reasonably well end to end, unless you want feedback on your design or feature set. The crowd testers are not going to read through product specification documents before testing; in fact, to encourage their creativity, you do not want them looking at your product documentation. Thus, your product needs to be working intuitively for them to start testing it and providing feedback. Also, it is important to ensure your core team has time to work on the feedback the crowd team provides, otherwise, the crowd is soon going to be demotivated and lose its association for you and your product, which is a big loss to you.

The "What" Factor: Typically, end-user-facing features—areas that require combinatorial testing, such as localization, compatibility, performance, usability, and accessibility—are areas where you get the maximum return on investment from the crowd.

The "How" Factor: It is important to identify a person who will drive the crowdsourced test effort. He is typically a project manager or a tester who understands the value of the crowd, can keep them motivated, is able to resolve their queries, who understands your product, and can serve as a good liaison between your core team and the crowd. It is important to empower such a person with all that he needs to succeed, including collaborative tools and infrastructure, such as private clouds, etc., because if he succeeds, your crowd succeeds, which means your product has greater chances of succeeding in the marketplace. Also ensure you get your stakeholders and project sponsor’s buy in into this entire effort, as unless you have that, you will not be able to go very far.

You are leading a session on crowdsourced testing at the upcoming STAREAST conference. For those who can't make it out to the event, what is the "big takeaway" you want attendees to get from your talk?

With the current competitive forces at play in the market, such as faster time to market and need for a reduced spend, it is becoming increasingly important to embrace newer and creative test techniques such as crowdsourced testing. Crowdsourced testing is further becoming relevant due to the need for: domain knowledge, testing on realistic end-user environments, and global distribution of products.

The crowdsourced testing landscape is huge and offers some very creative avenues for you to leverage the knowledge and inputs of the community to benefit your product, before it has been shipped to the end-users. Keep an open mind in evaluating this technique to see if it makes sense for your product, and if so, consider the options presented above to chalk your execution strategy. Start small with a pilot or proof-of-concept to win your stakeholder's buy in and also for you to be assured that the model works with your set of constraints before branching off to something big.