Crowdsourced Testing: An Interview with Rajini Padmanaban

[interview]
Summary:

Rajini Padmanaban is the director of engagement at QA InfoTech. Heather Shanholtzer had the opportunity to interview Rajini and learn a bit more about crowdsourced testing and find out why it is better than traditional testing in some projects.

Rajini Padmanaban is the director of engagement at QA InfoTech. Heather Shanholtzer had the opportunity to interview Rajini and learn a bit more about crowdsourced testing and find out why it is better than traditional testing in some projects.

Heather Shanholtzer: What is crowdsourced testing and why is it useful?

Rajini Padmanaban: The "crowd" is a diverse set of people from the community at large, who can be leveraged in testing and providing feedback on a product because of their end-user representation, domain knowledge, subject matter expertise, or software testing experience.

Crowdsourced testing is the process of using the crowd and employing a strategy to have them provide formal and informal feedback on the product under test. In my humble opinion, the most value from crowdsourced testing is reaped in the first three scenarios above, rather than relying on the crowd's testing experience alone, which you can largely get from your traditional testing methods. In the first three groups, the crowd tester is a more realistic tester than a trained tester.

The crowd can be engaged for monetary gains for the work they do, but this is not always the case. Pride of participation, non-monetary rewards, other accolades, sense of satisfaction, and to quench one’s personal or professional aspirations could be other drivers that engage the crowd.

It is often misinterpreted that crowdsourced testing is about engaging and working with a company whose business model is to pool a crowd of testers who get paid for valid bugs they report. Well, this is definitely one form of crowdsourced testing, but there are several other more important and creative manifestations to look at, to leverage the crowd's wisdom including:

  • Sourcing relevant people from within one's company, although they might not be directly working on the particular project
  • Building a pool of end-users gradually over time to provide feedback across various phases of product development (be it public beta users, dogfood programs, private betas covering Most Valuable Players, etc.)
  • Partnering or working with organizations and universities who have domain knowledge and subject matter

Heather Shanholtzer: What are some of the risks of crowdsourced testing?

Rajini Padmanaban: Crowdsourced testing is not a no-brainer solution that is free of risks and challenges. Risks exist in two categories: implementation and psychological. On the implementation side, unless you make an educated decision of what, when, and how to crowdsource, the effort can soon become chaotic, overwhelming, and randomizing for both the crowd and the team that is managing the effort. Fortunately, there are clear mitigation strategies to alleviate the risks, in the form of best practices to adopt.

One of the biggest risks is that the crowd is not contractually bound to you or your product, unless you work with a very private crowd where you sign a non-disclosure agreement (NDA) with them. In cases where you do not have an NDA, you want to be selective about what you want to crowdsource to be able to leverage the crowd's inputs but not compromise your intellectual property.

Also, it is important to understand under what circumstances the crowd typically succeeds and try to maximize on them, so as to minimize risks. In my experience, I've seen crowdsourcing succeed when: 1) A diverse crowd is chosen to test the product, 2) independence is maintained in their communication unless the product requires them to talk amongst themselves to promote knowledge sharing, 3) the product has a global user base requiring vast domain knowledge that is difficult to source and live user environments that are difficult to simulate internally, and 4) when tasks assigned to the crowd align with factors that motivate them.

On the psychological side, unless adequately addressed, the core test team (be it your internal team or a vendor or outsourced test team) may feel threatened by the presence of a crowd team. This not only demotivates the core team but also reduces the chances of success for the crowd, due to lack of the core team's support for the crowd test effort. The sponsor needs to ensure that everyone involved understands that the crowd is really a supplemental team to your core team—not a stand-alone team—with each having its core competencies. When this is established, a sense of empowerment flows in, where the core team feels more confident and will go out of its way to support the crowd test effort.

Heather Shanholtzer: Under what circumstances is crowdsourced testing a better option than traditional testing?

Rajini Padmanaban: To some extent, I've answered this question in my previous response on risks in crowdsourced testing. Typically, crowdsourced testing is a better option when the product has a global user base requiring vast domain knowledge that is difficult and ineffective to source full time and when it is difficult to simulate all of the end-user scenarios in house. The following are a couple of cases when the crowd is a very good resource to leverage:

  • Testing end-user-facing features where the crowdsourced team represents the end-user mindset and behaviors
  • Performance, localization, and compatibility testing that need to cover several end-user environments

About the author

Heather Shanholtzer's picture Heather Shanholtzer

<div class="techwell_curator_bio">

As Software Quality Engineering's director of publishing, <strong>Heather Shanholtzer</strong> oversees content generation and dissemination for <em>Better Software</em> magazine, TechWell.com, AgileConnection.com, CMCrossroads.com, and StickyMinds.com.

After nearly ten years of helping software professionals find their literary side, Heather is still excited to discover new voices and she relishes the opportunity to work with industry experts, practitioners, and craftspeople.

If you would like to write for one of our publications, email Heather at <a href="mailto:HShanholtzer@sqe.com">HShanholtzer@sqe.com</a>.

</div>

Upcoming Events

Sep 24
Oct 12
Nov 09
Nov 09