You also need plans that show how you will create the product, and how you will mitigate those risks through process, tools, testing, etc. So from my perspective, I was responsible for creating a test plan. You also have to make sure you have verification testing in place, so you look at the requirements and specs and test off of them, as most traditional testers would. If you notice in the FDA regulations, they talk about testing approaches like unit testing, system testing, functional testing, and integration testing. That helps testers who like to use different models of coverage because it is easier to sell using this kind of approach to other stakeholders. That was just our starting place though; we went far beyond based on what we discovered through our risk assessment process and for attributes and affordances that were unique to mobile devices. A lot of time was spent on failure modes and how the app could handle them.
We required validation and verification in two specific areas:
- Clinical studies: Highly trained radiologists recorded their findings on FDA-cleared products and on our mobile apps on devices, and demonstrated there was no appreciable difference in the findings.
- Technical bench testing: A third-party organization conducted display validation using the devices to generate test images.
Next, for auditing purposes, someone from outside the team needs to be able to select tests and run them and will also look at failures. They will open up a random bug report and follow it. Can they repeat it? What was done to fix it, and what was done to verify it was fixed? They will look at the entire system from risk assessment to plan to tests to failure report and retesting for traceability. What test failed? How did you know there was a failure? How can you tell whether it was truly fixed or not? What safeguards do you have in place to ensure the fix stays in the code? It's like any auditing situation: You have processes and safeguards in place and an outsider can have a look at what you are doing and understand them. Accounting firms do the same thing and I looked at the FDA guidelines much like GAAP (generally accepted accounting principles) and the FDA is quite permissive in exactly what you do, as long as it follows the spirit of the regulation. They don't demand one certain way of developing software because they understand that there are many acceptable ways of doing this well and technology can dictate certain practices. Their "lease burdensome approach" concept is really nice. I've worked with other agencies who weren't as open minded. I had one agency try to force a Best Practices model of software development and testing that was created by an accountant. It was so out of step with reality it was absurd. They had no clue about iterative, incremental methodologies, the agile movement, automated testing, let alone risk-based or exploratory testing. In contrast, I have found the FDA process to be much more reasonable.
JV: Was the testing more difficult in a mobile medical app environment as opposed to more traditional types of testing?