The New Ways to Break Mobile and Embedded Software: An Interview with Jon Hagar

 Noel: I read all these stories online of testers having to prove their worth and trying to justify their need in the software world. It's horrifying to think that testers aren't being regarded so highly when it comes to the prevention of keeping something like pacemakers from being hacked into or planes being brought out of the sky. Who do you think will be ultimately credited for giving testers the respected position within an operation that they deserve?
Jon: I might suggest that testing, or any single development effort, should never be considered "priority one." The problem with calling something a “number one” priority is that each group tends to think their job focus is undervalued and should be viewed as more important or a number one priority. I teach and talk with business analysts, proposal teams, system engineers, software engineers, hardware engineers, programmers, testers, QA, CM, and others. Each feels their contribution is undervalued. Being an agile supporter, I would advocate that each of these disciplines is of equal importance and value. Each group needs to focus on providing value and quality to the overall product and project. Project personnel and even managers both need to look for balance—and this includes testing.
Now, to the implied problem of testing being called “second class,” of lower value, being paid reflective of that perspective, and being perceived as a non-value-added “tax” on development, which some managers or other stakeholders might wish to be rid of: We hear the cry “Testing is dead,” not just in embedded and mobile systems, but with other software too. Who will change these perspectives? I think some kinds of testers may have an endangered skill. If all you can do is write a detailed test procedure based on requirements and run that test over and over at the end of the development of the embedded system, you might want to rethink your job, because your days may be numbered.
However, in the embedded and mobile world, if you:
  • have skill and knowledge in system, hardware, and software engineering
  • can provide valuable quality information during the development
  • can support the team in rapid development with, hopefully, a fast project closeout testing effort (measured in perhaps hours or days, not weeks or months) to tell stakeholders the device is ready to ship
… then, my answer would be we as testers control our own worth and respect. Of course, this is just my old, jaded opinion. In the cases of mobile and embedded software, I discovered that as a tester, the information that I provided to the whole team was valued and respected. In the book, I tried to capture the things (attacks) I did to get this respect.
Noel: You yourself ask a great question right in the first chapter of your book:
But why cannot we just use the testing common to other software and systems? In the 
next section, you will discover the differences. Testers must have additional attacks in their 
toolboxes with all of the differences and unique bug patterns. The tester may use attacks 
from the How to Break Software book series, but new attacks should also be considered.
Why can't we use previous testing mindsets, tools, and strategies? What about mobile, embedded systems provide a unique challenge that "common" or "traditional" testing methodologies just can't cover fully?
Jon: In part, the tools, techniques, strategies, and mindset of traditional IT testing continues to be applicable in the mobile and embedded software devices. Testers in the mobile and embedded space should not forget classic testing concepts. My book cites and cross-references to books and attacks by James Whittaker and others that were aimed at the PC/IT systems. I also discuss many test concepts and techniques that traditional testers will find familiar. But here again, the taxonomy showed error patterns which lead me to define attacks in areas that were different than the ones talked about by James Whittaker. A tester moving into testing mobile and embedded systems must be aware of these different error patterns and factors so they can start to update their mental models of where bugs hide.
For example, many mobile and embedded devices are very battery-dependent. How many IT software testers have ever thought about the ways to test battery performance from a software perspective? And yet we know that software can directly impact the battery usage on a mobile or embedded device, and users really care about battery life. The book lists and examines attacks on such areas of mobile and embedded devices, including the mobility itself, batteries, environments, network connections, resource limitations (CPU speed, memory number of apps, etc.), and device characteristics like screen size, user interface, hardware interfaces, etc. A tester moving from traditional IT/PC/large computer testing to the mobile and embedded space can quickly get ideas for improvements in their testing because of the different mobile and embedded contexts. Context matters.
Jon Hagar is a systems software engineer and tester supporting software product integrity, verification, and validation with a specialization in embedded software systems. For more than thirty years Jon has worked in software engineering, particularly testing/verification and validation. Embedded projects he has supported include the space shuttle, large rockets, spacecraft, and ground systems as well as testing new smart phones. Jon has managed and built embedded test labs, including supportive automation development. Jon publishes regularly with more than fifty presentations and papers in software testing, verification, validation, agile, product integrity and assessment, system engineering, and quality assurance. He teaches classes at the professional and college level.

About the author

Upcoming Events

Apr 17
Apr 17
May 01
Jun 05