Heather Shanholtzer: Is crowdsourced testing appropriate for all projects?
Rajini Padmanaban: No. Crowdsourced testing is not a straightforward solution for all projects. Intrinsic characteristics of the crowd, the product under test, and the project’s constraints play key roles in deciding whether or not crowdsourcing is going to be useful in a given case. Unless this is planned for effectively, bringing in the crowd adds a significant risk factor to the project, as discussed earlier. The following are some cases where you would be better off working with your internal core test team with atraditional testing model:
- Areas where you have a lot of moving pieces and dynamics that require a lot of internal collaboration
- Areas of very sensitive intellectual property
- Areas that are dependent on some internal hardware or software resources that are difficult to simulate externally, at least at the given point of time
- Test automation, test-driven development scripts, regression, integration testing, first levels of performance, security testing that require discussions with business teams to baseline metrics
Heather Shanholtzer: How does one get started implementing crowdsourced testing in his organization?
Rajini Padmanaban: Crowdsourced testing is an educated decision that needs to be made keeping in mind what, when and how to crowdsource. In my experience, the "when" factor is the most important, followed by "what," and finally "how."
The "When" Factor: It makes the most sense to crowdsource when your product is working reasonably well end to end, unless you want feedback on your design or feature set. The crowd testers are not going to read through product specification documents before testing; in fact, to encourage their creativity, you do not want them looking at your product documentation. Thus, your product needs to be working intuitively for them to start testing it and providing feedback. Also, it is important to ensure your core team has time to work on the feedback the crowd team provides, otherwise, the crowd is soon going to be demotivated and lose its association for you and your product, which is a big loss to you.
The "What" Factor: Typically, end-user-facing features—areas that require combinatorial testing, such as localization, compatibility, performance, usability, and accessibility—are areas where you get the maximum return on investment from the crowd.
The "How" Factor: It is important to identify a person who will drive the crowdsourced test effort. He is typically a project manager or a tester who understands the value of the crowd, can keep them motivated, is able to resolve their queries, who understands your product, and can serve as a good liaison between your core team and the crowd. It is important to empower such a person with all that he needs to succeed, including collaborative tools and infrastructure, such as private clouds, etc., because if he succeeds, your crowd succeeds, which means your product has greater chances of succeeding in the marketplace. Also ensure you get your stakeholders and project sponsor’s buy in into this entire effort, as unless you have that, you will not be able to go very far.
Heather Shanholtzer: You led a session on crowdsourced testing at the STAREAST 2012 conference. For those who couldn't make it out to the event, what was the big takeaway you wanted attendees to get from your talk?
Rajini Padmanaban: With the current competitive forces at play in the market, such as faster time to market and need for a reduced spend, it is becoming increasingly important to embrace newer and creative test techniques such as crowdsourced testing. Crowdsourced testing is further becoming relevant due to the need for: domain knowledge, testing on realistic end-user environments, and global distribution of products.
The crowdsourced testing landscape is huge and offers some very creative avenues for you to leverage the knowledge and inputs of the community to benefit your product, before it has been shipped to the end-users. Keep an open mind in evaluating this technique to see if it makes sense for your product, and if so, consider the options presented above to chalk your execution strategy. Start small with a pilot or proof-of-concept to win your stakeholder's buy in and also for you to be assured that the model works with your set of constraints before branching off to something big.