Not too long ago, I worked in finance. My company made fraud-detection software for banks. It was complex, had a direct effect on people's lives, and was carefully regulated. We had multiple audits every year, which generally fell into two categories: risk assessment and regulatory compliance. I am not a lawyer and your industry's regulations may vary drastically from mine, so for now, let's focus on risk-based audits. Most of my experience with risk assessments has been as part of the Service Organization Controls (SOC) audit.

These audits involved a series of interviews and document reviews designed to answer the questions:

1. “Do you do what you say you do?”
2. “Has your company taken the appropriate actions to reduce your customers' exposure to reasonable risks?”

Depending on your business, this may include a review of finances, physical security, network and computer access controls, policies, and other documentation. Some auditors may be interested in shadowing you while you work and asking you questions about what you're doing.

In our case, after the auditors collected the necessary information, they evaluated our practices against some standards and wrote a report with their findings and recommendations.

Three Lessons I Learned
1. The auditor is looking out for your customers. If you think of software as a formalized business process, then auditors are simply another type of tester. When I work with programmers and business people, I want my work to help them be better at their jobs. In the same way, the auditor wants to help you serve your customers better. The policies or regulations that the auditor uses for guidance were probably created because of situations that, while profitable for the company, may present an undue risk to your customers, investors, or the public. Try to keep that in mind as the auditor makes his recommendations.

2. Auditors don't dictate your processes. Think of the auditor as a coach. He will push you and maybe even challenge you, not because he wants to make your job more difficult, but because he wants you to grow. Take his words as advice. You probably know your company and its culture better than your auditor. If something doesn't sound right, ask questions to find out why the auditor has suggested you change your process. You may find that the advice is a lesson another company learned the hard way.

As you're talking to your auditor, you may discover that his background is in an area other than software—perhaps accounting. He may be unfamiliar with the nuances of various techniques. Our company's auditors wanted me to explain the testing phase of our agile project. I could have argued “We don't have a testing phase,” but the auditor would have heard that as “We don't test!” Instead, I explained how we are always doing some kind of testing. Our auditors could see the benefit when I explained specifically how we worked.

3. Know how you'll be measured. There's nothing like walking into a university exam unprepared, and the same goes for audits. Find out what laws and industry practices apply to your business and discuss them with your auditor. Your auditor may even have a self-assessment to help you identify areas of improvement before the audit.

Why Auditors Love Exploratory Testing
Auditors explore, too!
In the discovery phase of the audit, auditors are reading, listening, thinking, and asking questions about the information they encounter. They adapt to what they've learned. Sure, they may start with a checklist, but they're hardly confined to a plan. They analyze what they've learned and share it to help their customer make better decisions. If you re-read the previous four sentences with “testers” substituted for “auditors,” it sounds a lot like exploratory testing. Don't you think your auditor will support you when he realizes you work the same way he does?

Exploratory test documents tell a story.
Imagine you're an auditor visiting a company for the first time. You talk to a tester who methodically follows test scripts and reports which steps passed or failed. When you review the test results, how do you determine if the test was appropriate? How do you know how the test impacts the business? You would have a very limited understanding of the value of the test without also reviewing the system architecture, test design, and the decisions that were made after the test was completed. That's a lot of work, and you could only guess why this person did these things.

Now, imagine the same situation in which the tester's notes explain his thought process, what he wanted to test and why, what he didn't test and why, what he reported, and what decisions were made from the reports. This style of documentation gives a more complete view of the test. If I were the auditor, this is the kind of information I'd want. If the Sarbanes-Oxley Act applies to your company, your auditors will appreciate test documents that discuss how decisions are being made.

But my auditor wants extensive documentation!
Then give it to him! Following test scripts isn't the only way to document your testers' activities. Would he accept videos of the testers as they work, computer logs, screen captures, saved outputs, or some other evidence that the test was performed? An auditor asking for documents to prove that the test was executed may be more interested in verifying that you're performing the actions in the way that you say you do rather than in how effective those actions are. Our test policy stated that we would use a variety of techniques to evaluate software and provided an open-ended list of example techniques. As a general rule, I want my level of rigor to match my risk. If the project could result in people getting hurt, legal action, etc., I'm going to create solid documentation that can hold up in court.

That much documentation would be overkill for some projects, however. Remember the words “appropriate” and “reasonable” when you're thinking about your processes and risks. A short document explaining what you plan to test (charters) and how you'll evaluate it (oracles) along with a document summarizing your findings may be sufficient for less risky projects. My “test plan” and “test result” documents were usually less than ten pages combined for projects lasting two or three months. I also had an extensive archive of notes, videos, system logs, test data, and system outputs in case anyone had questions about the test results, but I never had an auditor ask for the details. If your auditor wants more, find out what he's trying to evaluate and provide information to answer his inquiry.

Making the Transition
Testers on your team are probably already exploring to some extent, so formally acknowledge it by mentioning exploratory testing in your procedure documents. To prepare for your next audit, include documents from exploratory test activities in your regular test documentation. If your auditor likes to shadow you, encourage testers to talk through their thoughts with the auditor. If some members of your team are uncomfortable talking about exploratory testing, consider using in-house training to help them be more confident in the skills they're using.

Exploratory testing can help you make the best use of your team's creativity, adapt to changes, and provide visibility into the decision making process. Don't miss out on these benefits because of an adversarial stance with your auditor. Reach out to him as a trusted partner; it just might make your audits a pleasurable experience!

References
Sarbanes-Oxley Risk Assessment Guide

SOC Audit Toolkit