Under what circumstances is crowdsourced testing a better option than traditional testing?
To some extent, I've answered this question in my previous response on risks in crowdsourced testing. Typically, crowdsourced testing is a better option when the product has a global user base requiring vast domain knowledge that is difficult and ineffective to source full time and when it is difficult to simulate all of the end-user scenarios in house. The following are a couple of cases when the crowd is a very good resource to leverage:
- Testing end-user-facing features where the crowdsourced team represents the end-user mindset and behaviors
- Performance, localization, and compatibility testing that need to cover several end-user environments
Is crowdsourced testing appropriate for all projects?
No. Crowdsourced testing is not a straightforward solution for all projects. Intrinsic characteristics of the crowd, the product under test, and the project’s constraints play key roles in deciding whether or not crowdsourcing is going to be useful in a given case. Unless this is planned for effectively, bringing in the crowd adds a significant risk factor to the project, as discussed earlier. The following are some cases where you would be better off working with your internal core test team with atraditional testing model:
- Areas where you have a lot of moving pieces and dynamics that require a lot of internal collaboration
- Areas of very sensitive intellectual property
- Areas that are dependent on some internal hardware or software resources that are difficult to simulate externally, at least at the given point of time
- Test automation, test-driven development scripts, regression, integration testing, first levels of performance, security testing that require discussions with business teams to baseline metrics
How does one get started implementing crowdsourced testing in his organization?
Crowdsourced testing is an educated decision that needs to be made keeping in mind what, when and how to crowdsource. In my experience, the “"hen" factor is the most important, followed by "what," and finally "how."
The "When" Factor: It makes the most sense to crowdsource when your product is working reasonably well end to end, unless you want feedback on your design or feature set. The crowd testers are not going to read through product specification documents before testing; in fact, to encourage their creativity, you do not want them looking at your product documentation. Thus, your product needs to be working intuitively for them to start testing it and providing feedback. Also, it is important to ensure your core team has time to work on the feedback the crowd team provides, otherwise, the crowd is soon going to be demotivated and lose its association for you and your product, which is a big loss to you.
The "What" Factor: Typically, end-user-facing features—areas that require combinatorial testing, such as localization, compatibility, performance, usability, and accessibility—are areas where you get the maximum return on investment from the crowd.
The "How" Factor: It is important to identify a person who will drive the crowdsourced test effort. He is typically a project manager or a tester who understands the value of the crowd, can keep them motivated, is able to resolve their queries, who understands your product, and can serve as a good liaison between your core team and the crowd. It is important to empower such a person with all that he needs to succeed, including collaborative tools and infrastructure, such as private clouds, etc., because if he succeeds, your crowd succeeds, which means your product has greater chances of succeeding in the marketplace. Also ensure you get your stakeholders and project sponsor’s buy in into this entire effort, as unless you have that, you will not be able to go very far.
You are leading a session on crowdsourced testing at the upcoming STAREAST conference. For those who can't make it out to the event, what is the "big takeaway" you want attendees to get from your talk?
With the current competitive forces at play in the market, such as faster time to market and need for a reduced spend, it is becoming increasingly important to embrace newer and creative test techniques such as crowdsourced testing. Crowdsourced testing is further becoming relevant due to the need for: domain knowledge, testing on realistic end-user environments, and global distribution of products.
The crowdsourced testing landscape is huge and offers some very creative avenues for you to leverage the knowledge and inputs of the community to benefit your product, before it has been shipped to the end-users. Keep an open mind in evaluating this technique to see if it makes sense for your product, and if so, consider the options presented above to chalk your execution strategy. Start small with a pilot or proof-of-concept to win your stakeholder's buy in and also for you to be assured that the model works with your set of constraints before branching off to something big.