Why You Need to Do More Than Just Test Requirements: An Interview with Jon Hagar


Then you look at a lot of the reviews in social media that sprang up at the same time. The users can quickly brand an app as good or bad, and it can basically kill an app if enough reviews get posted online in the app store that say, "Hey, this thing doesn't work" or "This doesn't work on my particular platform."

JV: That’s such an interesting concern. That instant review, the instant critique.

JH: Yeah, and you see things posted in there that where I came up with my book was by reviewing a lot of the public available information by places like app stores and news reports across the mobile smart-phone space and the embedded space, and I tried to get a handle on the kinds of errors that were, I'll say, being found by users out in the field very commonly. From that, followed along with Dr. James Whittaker's work of attack-based testing, with the idea that you not only want to work and test the functionality of some software, show that it does the requirements is the classic one, but that you also want to come along and really try and show that something doesn't work. The break-it philosophy.

JV: I was wanting to get into that; why do we need to break mobile and embedded devices?

JH: My answer is that if we, as the test community, don't, the users will, and we see the bad reviews. In some of the mobile and embedded space, we see various horrors stories, even lawsuits, from systems not working correctly. So, there's in my mind a fair amount of the need to have the right amounts of testing. I'm not saying that every mobile smart phone app out there needs to be tested at the same level as, say, the space shuttle was tested, but you need enough testing early on so that when you get it out in the field, hopefully rapidly, that you don't get those bad reviews, because otherwise, you're going to probably not be around to have another go around of changes.

It's a balancing game between the level of cost and schedule and how much testing, where I think, unfortunately, some places don't do enough testing.

JV: From some of the folks that you talk to, are the reviews really making them want to double down their testing? Is there a sense of urgency out there?

JH: I'm not sure the urgency is there yet. I think some of the mobile app developers have started to realize they need more. This was what we saw back in the dot-com days and the PC world where you could get away with almost anything. As the users become more sophisticated and the social media increases, I think a lot of the people will start going, "Gee, we need that right level of testing," and again, there's no one size fits all of that.

JV: What are some things traditional programmers misunderstand about mobile?

JH: I mentioned the one about batteries. That's one I found quite a bit and offer up a bit of specific attack on. The other one that's cropped up quite a bit recently and scares me quite a bit is the whole security end of things, in everything from mobile banking to other fields. I was watching news pieces where people were walking around stores and being tracked; Is that what they want?

There's a whole set of security issues; the limitations of the devices themselves. You have small screen size. That's a limitation. You have, relative to a PC, smaller amounts of memory. Yes, there's a lot of memory, but smaller amounts of memory and storage capability. I think one of the things a lot of the PC people forget, because they're used to just very high bandwidth in a direct hard-line connection, is you have the connection to the mobile service provider, which can come and go. You can get low bandwidth, high bandwidth; it can change as you move around. How does that effect the app and the software? It becomes an issue that everybody needs to think about.

JV: There's another body that the thing needs to communicate with that might not have been there in the PC world.

JH: Yeah, where you get it to pop out and lose some data or something like that.

JV: How about other major security issues?

JH: Well, and it's evolving quickly. In the book, I had some attacks. I think those are still good and valid. Your basic login signature kind of thing, but things are changing rapidly. You have the biometrics concepts that we see a little bit with some of the Apple products, where you have fingerprint recognition. I think those things are going to both help security but then introduce a new level of, I'll call it, concerns for testers. If you've got critical data that you're exposing, that may not be what you want.           

You also see spoofing. There's something known as GPS spoofing that's very sophisticated but nonetheless possible. Do you want to have yourself hijacked when you're driving your car?

JV: That's a pretty major concern, I would suspect, that people would have.

JH: We know from reports people have cracked into cars. We know people have cracked into pacemakers, another kind of embedded device. Again, a little bit of knowledge for me just makes me very scared, and I've been studying this a fair amount, but I still have an awful lot to learn. I think for those of us in the mobile security community, it's going to be a lot of years and a lot of hard work.

JV: You have a chapter in the book on smart hand-held mobile systems. What is a smart hand-held mobile system? What would constitute something to be smart?

JH: The most obvious one are the Androids and the iPhones and those kinds of things that I think probably a lot of people have, but there's also other things that are out there that maybe we don't think about as much. A lot of the cars these days come with a network online and information and entertainment systems built right into the car. A car is certainly mobile. It's somewhere in between a phone and an embedded device.

I recently flew the new Boeing airplane. It's got an online information system.

About the author

Jonathan Vanian's picture Jonathan Vanian

Jonathan Vanian is an online editor who edits, writes, interviews, and helps turn the many cranks at StickyMinds, TechWell, AgileConnection, and CMCrossroads. He has worked for newspapers, websites, and a magazine, and is not as scared of the demise of the written word as others may appear to be. Software and high technology never cease to amaze him.

Upcoming Events

May 04
May 04
May 04
Jun 01