GA: Because you have no idea what was doing, what brought it on, and no detailed statistics on what they've done in terms of troubleshooting. It becomes a needle-in-a-haystack trying to figure out what the issue is.
GA: I typically look at the help-desk forums that people have submitted tickets for issues, on-site issues, or app issues, and also how they detail their beer merges or their beer edits in our system. If I find that those two are pretty high level, then I'll reach out to them and say "Would you like to be part of our super-user group?" To be honest with you, everyone I've reached out to has accepted and is really excited to do it. I think you find the right people that are dedicated and are fans of your platform, they make the best testers because they really care about what they're talking about as opposed to someone who just says they want to do it and then you never see them for like two months.
JV: These people are really going out of their way at the beginning to write detailed responses and you're just getting them along.
GA: I love to hear bug reports and I love to fix things, but one of the most irritating things for me is when people just say, "It doesn't work." I think that helps nobody and I think when you're looking to help to build your platform and help you build your data and help you fix bugs, you want those dedicated people that spend the time to show you exactly what browser they're using, what processes they did, screen shots, and stuff like that. That helps everything a lot. Synchronicity and then go.
JV: I was just going to ask you: a pretty common dilemma facing testers in mobile is testing on all the various platforms.
GA: Oh yeah.
JV: So how do you go about doing it? Do you assign people in your group? Does someone have a Nexus phone and you have them test everything on that? What is the process like for that?
That solves the problem development, but the problem also exists on the other side of the table from the testing perspective. We have a lot of tools to help us get feedback. One is TestFlight, which, up until now was supporting Android and iOS, but now is only iOS and that allows us to basically build our IPA—no pun intended for the beer. It happens to be the name of a file, but what it does is send that file out automatically to all our testers and it actually integrates with our platform so when crashes happen, it actually sends information back to TestFlight, so we can see in real time what's happening with the errors that are occurring.
JV: That's very handy.
GA: That helps significantly. All of our beta users will have an opportunity to sign up for TestFlight when they first join the super-user group. They have an opportunity to see the iOS version or the Android version. TestFlight, at least it used to before it dropped the Android support, would separate those out, so you could upload an ADK or upload an IPA and they would basically just filter out. The beta user downloads them and they get real-time analytics on how they're doing. Up until now, we've obviously couldn't use feature for TestFlight, but I've discovered a pretty new product that's out there called TestFairy.
JV: TestFairy, OK.
GA: It's specifically for Android. Kind of a little bit new in the process, but what I love about it, it allows users to actually capture their screen in terms of video of how they're using the application and what their processes are for producing a bug. Now, with Android, when you use Jelly Bean or higher, the logcat isn't available for users. The actual log of errors is not available unless your phone is rooted. It makes it really difficult for users to do actual de-bugging, because if they hit an error or they have a non-responsive app, it pretty much dies and you have to ask them to replicate that process from like "How do I do it again, so I can see it on my screen when I'm plugged in to my console so I can see what the errors are.”
What TestFairy does, it actually installs a listener on top of your app. It actually records the video screen and how they're interacting with the app. I've never seen something like this before. I used it for about a week, two weeks now and it's very early in a beta phase. It's extremely helpful for users that are not very articulate with how they've actually gone through their process of finding a bug.