Mukesh Sharma and Rajini Padmanaban of QA InfoTech sit down to talk about their upcoming book, Leveraging the Wisdom of the Crowd in Software Testing. In this interview, Mukesh and Rajini also talk about the benefits and future of crowdsource testing.

 

Cameron Philipp-Edmonds: Today we have Mukesh Sharma and Rajini Padmanaban and they are going to be speaking to us today about their new book “Leveraging the Wisdom of the Crowd in Software Testing.” Thank you so much for joining us today guys.

Mukesh Sharma: Thank you.

Rajini Padmanaban: Thanks Cameron.

Cameron: All right, to get things started can you tell us a little bit about yourselves.

Mukesh: Sure, I am Mukesh Sharma. I am the founder and CEO of QA InfoTech an independent quality assurance and testing company that I founded in 2003 with the intent of to widen impartial, unbiased and independent testing services.

Rajini: Hi everyone, this is Rajini. I’m based out of Bangalore, India and I’m a senior director of testing engagements at QA InfoTech.

Cameron: All right, let’s start talking about your book here. What led you to both of you writing this book?

Mukesh: Well our practical experience and knowledge of crowdsource testing and lack of any good book on this subject encouraged us to write this book and the goal is to be able to share our knowledge with the community at large. Crowdsource testing is very near and dear to us, in fact we’ve been advocating this over the last seven to eight years including presentations and tutorials that we’ve done even for STAR conferences. We’ve been leveraging crowdsource testing in-house on a number of projects that needed to be tested on multiple smart devices. We’ve also built a portal called BugMoney.com that helps us in connecting with the external users or crowdsource testers to test for our clients’ products.

Cameron: Okay and you said at the beginning that there’s not a whole lot of books, good quality books, on this subject and I have to agree with you on that because there’s not a lot of knowledge even what crowdsource testing actually is. In common terms, can you describe what crowdsource testing is?

Rajini: Sure. If I were to put it in very simple terms it’s nothing but bringing in the community at large to test for your product and application. When I say test I’m not necessarily talking about testers. Ideally you don’t want testers coming in to evaluate your product, you really want people like end users, domain experts, or potentially the person could be a tester too. From our experience we see there’s a lot more value in bringing in an end user or a domain expert compared to that of a tester in testing of your product.

Also, if you see crowdsource testing can really be done with internal employees in your own organization or you could bring in the external crowd. It’s a phenomenon where you can really apply it across companies of different sizes, across companies of different technologies, you could be a services company, you could be a product company. It’s really a very versatile technique to bring in to evaluate your product. And one main thing to keep in mind is money is not what always motivates the crowd. The crowd could really be coming in to test your product based on their association with your brand, the fact that they’re going to transparently look at your product before it hits the market. There could be multiple factors that really motivate the crowd, but in very simple terms it’s nothing but bringing in the community to evaluate your product and provide feedback.

Cameron: All right, fantastic. That’s a great definition. Now what makes this book worth reading and who really is the target audience?

Rajini: Sure, I’ll probably take this one. As Mukesh rightly said in one of his, in his first response, as of today we don’t really have a good quality book on crowdsource testing. Also speaking there’s a lot of ambiguity still in the industrial community on what crowdsource testing is, as you rightly said you had that question to define crowdsource testing.

There’s one belief that it’s nothing but working with the company who in turn brings in the crowd to test your product, so that’s one form of it. That is also crowdsource testing but there’s several other forms as I just mentioned, it could be the internal crowd, it could be the external crowd, it could be crowd that comes in on beta testing assignments, bug bashes, field testing. There’s so much more to it. There’s a lot of step-by-step approach that you can really follow and implement in crowdsource testing.

That’s what we really wanted to bring in this book, we wanted to really start talking about what crowdsource testing is and the flow of chapters help a person really implement crowdsource testing, talk about the limitations, challenges, solutions so really give an end-to-end perspective of the topic to the reader. Someone who comes in who wants to understand how can I build a career in crowdsource testing, what are the trends, what does the future look like? Really give the reader a very well-rounded and full knowledge and that’s what we’ve really tried to address in this particular book.

We’ve talked about several case studies, we’ve talked about several examples from QA InfoTech itself. We’ve also talked about a lot of examples from Microsoft for instance. Here I should say the testing director of Microsoft Skype division, a gentleman by name Ross Smith he’s been a significant support for us, significant source of encouragement for us to write this book and he’s shared a lot of examples from Microsoft.

In terms of target audience if you see this book really is a good fit for testing groups that are building a crowdsource testing effort, any individual who is working on this effort, for the community at large anyone who wants to understand what it is and how to implement the book really is a good fit for several of them out there.

As Mukesh again said I think when we researched initially on this topic as of today there’s literally one or two books on this topic. There was one on Barnes & Noble that we saw and the lady herself was mentioning that more than really a book this is just a collection of online material so with all of this we said ‘this will be a great book to release and this will be a great year to get it out in the market as well.’

Cameron: Right and that sounds very comprehensive and it really seems like there’s something for really everyone.

Mukesh: Absolutely.

Cameron: You guys have both written both for StickyMinds, TechWell and other industry sites and what’s different with those is that they’re smaller pieces of work, they’re much more digestible chunks of writing, so what are some of the limitations and benefits of writing extensively on a subject for an entire book rather than just those small, short pieces?

Mukesh: You’re right, given that we know what it takes to write blogs and articles, which are fairly short and as you pointed out digestible and also faster to write. A book has its own set of challenges. It takes a significant amount of time and commitment to author a book even more when you coauthor because while you get to load balance and write individual chapters to finish faster the wavelength of the authors need to match, the articulation style needs to match. A lot of discussion and brainstorming is also very important.

The overall process of a book does not end with just putting down the information you have and you know. The sequence of chapters is important; you have to assume readers of varied levels of knowledge because you’ll have all sorts of readers. You need to be able to present your content according to the type of readers that are going to read it. This way you are not overwhelming a rookie, somebody who’s got a lot of experience, somebody who’s got no experience on crowdsource testing. There is something everybody.

The knack of keeping the reader engaged throughout the book is essential otherwise you will lose your readership very soon. On the positive side a book gives a lot more real estate to bring all your thoughts on a particular topic. This is challenging in a blog or a paper wherein you have limited space and you’re sharing all the information with reader in that limited space. The learnings from my first book on software testing was very useful in writing this one and I’m glad Rajini and I aligned very well on this book.

The next critical step is finding the right publisher who values your effort and can empower you and the book to succeed in the marketplace. We started writing this book in September of 2013 and in January we floated our idea to a few publishers and were very excited with the response that we received. Within a couple of weeks of floating our idea we were in active discussions with CRC Press Taylor Francis and the book is currently slated for release by them in September or October of 2014.

Cameron: All right, fantastic. You talked about the benefits of being able to write on the real estate of a book. To transition here what are some of the benefits of crowdsource testing?

Mukesh: Well Rajini has talked about crowdsource testing and I would say that crowdsource testing is a very effective supplement to the core testing efforts in a quality implementation. It’s not something that you would execute by itself, you need to have your core testing efforts in place and then use it as a supplement. When planned for well it brings in a wide range of resources from different backgrounds with different domain expertise to help you for a variety of your testing needs.

Like I said more importantly it brings in end user representation and domain expertise which is often difficult and expensive to source and retain within your organization. Given the diversity needed in a testing effort of late with increased user, social, mobile networking and global launch of products crowdsource testing is becoming more of a necessity. You’re launching a product on one of these smart devices, iOS or android, and that is going to be used across the world you want to make sure that it works in different networks, on different devices. What better than being able to leverage your pool of crowdsource testers that are geographically distributed across the world and you know what they have and what they are testing on and you can rely on them for the feedback that they provide.

It allows you to get more test coverage within the limited time and cost constraints a team works in. Testing in realistic end user environments like I said let’s say a particular network or a particular device with a particular OS is having a problem you won’t know until you get a user to test it in that environment. Then there’s team productivity, distributed quality effort, subject matter expertise, partnerships in some SMEs like association for blinds for accessibility are some of the important key words that come to my mind when you ask for benefits of crowdsource testing.

Cameron: Now you talked about diversity and unfortunately sometimes with crowdsource testing you don’t always get the highest level of professionalism, you don’t always get exclusively professionals who are testing your product. Is this a problem or is it an opportunity?

Rajini: Good question, in fact in our opinion it’s a perfect opportunity and a blessing in case of crowdsource testing. You really want non-testing professionals as well to come in and evaluate your product and provide that feedback like how an end user would react for when the product actually hits the market. That said there is a growing concern in the marketplace that there are a lot of amateur players as well coming in. Players want to come in with the hope of making some quick money doing some crowdsource testing, a situation where they genuinely don’t care for the product that they’re evaluating.

This is where the team that is putting together the crowdsource test effort it has a very important role to play. The team has to make sure it identifies the right crowd that is connected and providing genuine feedback especially since a bunch of these crowd testers are going to be working remotely. Yes it is a blessing but it has to be a careful blessing which is exercised diligently.

Cameron: Now, you also talked about the different mobile iterations and hardware iterations and all across the various networks and what not. A lot of people look at that and since there are all those iterations, crowd sourcing is almost a necessity to keep cost down and make it cheaper and easier and get a wider range of testing going on. Then there are other people who are against crowdsource testing because they think it’s kind of an easy way out because you’re pushing the work onto someone else. What do you have to say to people like that?

Rajini: Sure, whether it’s for reasons of time or money saving it is indeed true that crowdsource testing has gained popularity because of the growth in mobile computing. A variety of devices has Mukesh now just mentioned in one of his responses, a variety of devices, platforms, networks that an application needs to be tested on … the world has grown leaps and bounds and it’s becoming increasingly difficult for the testing … to have all of these in-house. Even if you have them in-house they’re going to become obsolete very soon, how am I going to maintain them on an ongoing basis, these are all very practical challenges.

Whereas with crowdsource testers the whole testing is now able to cover all of these variables it needs. Additionally some of these people also turn out to be real users or domain experts because of which the power of the feedback that’s coming out is really unparalleled.

That said it cannot be looked at as an easy and a cheap way out. As I mentioned in my previous response if the right decisions are not made around what to crowdsource, when to crowdsource, how to do it the results really can turn out to be catastrophic both from a project implementation as well as a team morale standpoint. These are some of the points or chapters that we’ve covered elaborately in the book as well.

Cameron: You’re right and I think it’s very true that a lot of people don’t understand the depth which crowdsource testing actually brings and the fact that there’s a lot to look at and it’s not as simple as just saying ‘all right here crowd, go ahead and take care of this.’ There’s a lot more to it, there’s a lot more testing and studying that has to go on to make sure that you get the right data and the right information that you want and need.

Now, You mentioned earlier how it’s becoming more and more popular and I know you covered this in your book so I don’t want to give away too much but what do you guys imagine is the future of crowdsource testing?

Mukesh: Well in my view the future is very bright. We’ve discussed this in detail in one full chapter in the book. We’ve talked about it from four different standpoints which are market, technology, business, and the user. If I were to list some core trends here the number of crowd testers will continue to grow significantly, we’ll see a trusted and differentiated crowd emerge soon. You know how you can go on a particular website and say here is how rate this particular product or this partuclar vendor. Let’s say if you were to use, there are growing number of platforms wherein you can go and buy a particular service, you know whether this vendor is forthright or not. That is how we believe that this is going to grow.

We’ll see a trusted and differentiated crowd emerge soon. Softer service level agreements will become more prevalent; careers around crowdsource testing will become more prominent. People will say I am not going to take one job I am in fact going to work in the time that I prefer to work and with the company that I want to work and I can work with multiple companies. We expect that to work too.

A crowd tested product will have better market acceptance. Today, if you were to tell you end users that we’ve got our product crowd tested they really don’t know what it means until they see the quality. End users will become more vocal in sharing their feedback. When we see growth in all the four segments that I mentioned a whole new ecosystem around crowd testing will emerge building a very strong base for it in the industry for years to come. Definitely, exciting times ahead and we’re happy to be publishing this book this year which we believe is a perfect time.

Cameron: All right, fantastic. You talked earlier about how you’ve learned from publishing a book previously that helped you with this book and you guys talked about your writing here at TechWell and on StickyMinds and the last time we talked Mukesh a couple of months ago you mentioned that QA InfoTech puts great emphasis on encouraging employees to write and share their extensive knowledge. Can you remind listeners why you encourage QA InfoTech to produce industry writings?

Mukesh: Absolutely. I strongly believe in sharing what you know and that it helps you grow even better. Whether it is speaking or writing on testing topics such opportunities shape a person’s articulation skill overtime, further it brings in more creativity as the individual works on bringing out his or her thoughts on a certain topic, attempting to contain within a given time or a space limit. All of these are great skills that a tester can leverage in his day-to-day job. In addition it gives them a chance to be in touch with what is happening in the industry as well as contribute back to the industry.

From a holistic individual company as well as industry growth there are several benefits of encouraging employees into evangelistic activities and I’m happy we are doing that in our company. I also believe that it allows us to go out there and tell the world that we are thought leaders in testing. I can tell you that it has helped in positioning ourselves as thought leaders in software testing, I’m also happy that I myself along with people like Rajini are doing it on a regular basis so we are leading from front for our team at QA InfoTech.

Cameron: All right, fantastic. Now Rajini how has being encouraged to write influenced and benefited you and your career? Mukesh kind of covered how it benefits QA InfoTech how has it benefited you personally?

Rajini: Yeah, awesome question again. Firstly a large chunk of Mukesh’s previous response really holds good for this question too. I started writing in my earlier company around 2006 or so, but it was not very regular at that time, it was quite sporadic. When I joined QA InfoTech which is back in Jan. 2010, I recall Mukesh calling me in one of my very first one on ones that I should start writing blogs for the company. To be honest Mukesh you’re right here with me I was really overwhelmed when you said that to me back then, but now four and a half years later this is one of my activities I really look forward to.

I still continue to write a couple of blogs every month for our QA InfoTech website you can check them out on QA InfoTech blogs. As you previously mentioned I write regularly for TechWell and StickyMinds, because personally for me to write more, I  read more. I take the time to understand what’s happening in the industry, whether it’s tools, technologies, processes, what’s happening to other organizations. All of this reading has really definitely helped me mature in my role.

To give you an example here back in 2012 I did a conference presentation for STAREAST and really that was one of my first public speaking experiences. Since then I’ve spoken at multiple places including tutorials and keynotes and I attribute a lot of this confidence to the writing opportunities I’ve had. Finally, it gives me great networking opportunities with wonderful people in the industry including people like Cameron which is awesome.

Cameron: Thank you.

Rajini: Yeah.

Cameron: All right, my last question before we wrap things up here, should we count on another book to come from the brilliant minds at QA InfoTech?

Mukesh: Well we are honored that you asked this. Writing a book as you would know is no easy task. We definitely want to encourage our employees to start off with writing in the form of blogs and articles which is what they are doing also and then for those that show promise we may down the line consider a book too. If the potential is there we are happy to empower them with what they need to move ahead. As for Rajini and me we may also be looking at other topics in the future but it is too early to comment at this time.

On the same note though I do want to thank CRC Press and Taylor & Francis again for making this book a reality. We are very excited and we can’t wait for September to see this book in the physical and digital stores.

Cameron: All right, fantastic. Once again that book is “Leveraging the Wisdom of the Crowd in Software Testing.” Once it hits shelves in September or October please check it out and once more this was Mukesh Sharma and Rajini Padmanaban and thank you so much for joining us today.

Mukesh: Thank you Cameron.

Rajini: Thanks Cameron.

 

Mukesh Sharma: As founder and CEO of QA InfoTech Worldwide, Mukesh is responsible for the company's vision and leads the organization's worldwide operations, marketing, sales and development efforts. He founded QA InfoTech with a vision to provide unbiased Quality Assurance (QA) testing solutions and has grown the organization to five Centers of Excellence and over 750 employees in 10 years.

Mukesh began his technology career with DCM Data Systems as a software engineer, and then worked at IBM Corporation, Quark Inc., Gale Group and Adobe Systems in software engineering and testing roles. He has a passion for excellence, an eye for detail and commitment towards customers that has enabled QA InfoTech to stand apart in the exceedingly competitive software testing industry. Under his guidance, the company has contributed to various innovative quality assurance and test-automation solutions.

Mukesh holds a Master of Science and Technology in Information Systems from Birla Institute of Technology and Science (BITS), Pilani, India, as well as a Master of Engineering in Engineering Management from University of Colorado, Boulder, USA.

 

Rajini Padmanaban: As Director of Engagement, Rajini leads the engagement and relationship management for some of QA InfoTech's largest and most strategic accounts. She is also involved in test evangelism and thought leadership activities, such as blogging on test trends, technologies and best practices; building the test brand for QA InfoTech and generating ideas for service enhancements.

Rajini has more than twelve years of professional experience, primarily in the software quality assurance space. Over the years, as part of Polaris Software Labs and later at Disha Technologies, Aztecsoft and MindTree, she has been in various client-facing roles including project management, engagement management and QA pre-sales for leading ISVs such as Microsoft. Her primary areas of expertise are technical account mining and customer relationship management, both of which help her take existing strategic accounts to new heights.

Rajini holds a bachelor’s degree in Commerce and a master’s degree in Computer Applications, from University of Madras, India, where she graduated at the top of her class in both degrees.

Podcast Music: "Han Solo" (Captain Stu) /CC BY-NC-SA 3.0