Now that the Worse than Worst Case Has Happened

[article]

Airplanes turned into suicide bombs. Skyscrapers falling down. Invisible airborne death particles. And then...well, we don't know what.

Who's doing QA for the American dream?

The company where I manage software quality assurance wasn't in the World Trade Center, but close enough that six weeks later we were still operating out of disaster recovery sites while we waited for our high-speed telecommunications lines to be repaired. Now we're back in our building, looking out over a mass grave and breathing the smoke from the still-smoldering ruins.

QA professionals get paid to imagine bad stuff happening. We envision everything from an invalid data type being entered into a field, to a replication link failing, to a natural disaster causing servers to tumble down off their racks. We take a sort of mischievous pride in coming up with scenarios that scare end users and senior management. Yet the terrorist attacks on the World Trade Center and the Pentagon, the anthrax infections, and the subsequent threats of more mayhem to come have been far, far worse than anything our most flamboyant Y2K prognosticators ever conceived of.

Software testers and QA engineers in New York's financial district are still struggling to cope with the horrific events that have engulfed us. We never were a particularly close-knit group¡Xour companies compete fiercely with each other, and we work long hours¡Xbut since 9/11 the ordeals we've endured have motivated us to reach out to each other more through professional associations and online discussions. (At least it helps pass the time and calm the nerves when we're evacuated from the subway again for a ¡§police investigation¡¨ by those guys in hazardous materials suits...)

There are many people who work in quality assurance for whom it's just a job. They execute test cases, record the results, collect their paychecks, and then go home and sleep peacefully. Then there are a few for whom it's not just a job, but more of a calling. On some profound emotional level, they really care about keeping the systems they monitor working properly.

I'm not a mental health expert, but in the weeks and months after 9/11, I noticed among my professional acquaintances throughout the U.S., that it is these dedicated, zealous colleagues who seem the most severely distraught. As time passes and bad news and hardships become routine, they continue to suffer from anxiety, depression, self-doubt, or a sense of futility. They're asking themselves "What's the point of testing release 3.24b when the simple tasks of everyday life could put me in mortal danger?" Or, "How can I go on designing worst-case tests when reality turns out to be so much worse?"

As many QA gurus have noted, QA engineers can't put the quality into a product if it wasn't designed well in the first place. Nor are we likely to be asked to formulate U.S. foreign policy or military strategy. Yet as we confront the probability of future attacks by a wily, ruthless enemy, we can calm our nerves, strengthen our sense of purpose, and truly serve our country if we remember who we are and what we have to offer.

QA professionals are trained to assess risk analytically and methodically. We know how to devise plans that probe for weaknesses and defects in critical systems. We routinely create procedures to close gaps in communication and ensure reliable, repeatable performance of complex tasks. When something goes wrong, we investigate thoroughly and base our conclusions upon data, not nervous speculation. We have the skills and experience to help transform free-floating paranoia into a focused, disciplined defense effort.

Among other things, Ground Zero has now become a tourist attraction. We natives are getting used to being asked all kinds of questions by sometimes weepy, sometimes furious camera-wielding visitors as we go about our business. We've had to pull ourselves together quickly to help our organizations survive. Based upon what we've seen and heard and tried ourselves since 9/11, here are some suggestions for QA colleagues who may have succumbed to paralyzing malaise¡Xand also for those who feel fine but may want to make a greater contribution to the cause.

In your workplace, be proactive. No matter what your official job description might be, consider the needs of your organization as a whole. If your organization is just too large, focus on your own projects. Contact colleagues in other departments. Ask questions. Offer suggestions to appropriate managers. At meetings, remain calm and professional. Create your own agenda:

  • Research your project's or organization's business continuity plans. Determine the extent to which they protect people, operations, property, and data. Inquire whether the plans include a scenario describing procedures to be followed if the Internet becomes unavailable for a week or more. Develop a testing strategy to verify whether the plans actually accomplish their objectives.
  • Examine the data architecture of your project or organization. Identify the most frequent points of failure and the weakest security areas. Find out who is responsible for troubleshooting outages and breaches, and how they handle different types of problems. If necessary, provide them with information on integration testing, stress testing, performance monitoring, or security audit services.
  • Evaluate your project's or organization's training programs. Under the current circumstances, a major goal of training should be to establish redundancy of knowledge. For every job, there should be more than one person who knows how to do most of it¡Xand is authorized to perform the task. This is especially important for workflow software: data should never get stuck in the system because the only user who executes a certain step is suddenly unavailable. Alert the training director and project manager to any shortcomings you discover.
  • Audit your project's or organization's system and test documentation. If you can't do this because of security restrictions, focus upon the documentation for your own team's assignments. The documentation should be comprehensive and enough to enable an entirely new team to take over a project, perform maintenance, and deploy enhancements within a week.
  • Conduct the same type of audit for your project's or organization's process documentation. Based upon the written procedures, it should be possible within a week for new staff members to add users to the systems, assign appropriate permissions to databases and operating systems, and take whatever steps your IT group follows to roll out new versions of the software you develop.
  • Review your project's or organization's practices for archiving source code. Make sure you have the current version of the production source code for every software product stored in a secure location. Encourage developers to archive their work-in-progress every day, or as often as possible, considering the work habits of multideveloper teams. The archive should include databases used for requirements gathering, change management, and bug tracking, and all project and process documentation.
  • Evaluate the configuration management policies your project or organization has devised. Determine whether there is a standard desktop/laptop/wireless configuration that all users are expected to install (or a list of approved alternatives). Find out if there is a comparable standard for software developers and testers, both for in-house staff members and for external consultants. Inspect the configuration, the administrative procedures, and the audit process for security holes. Then re-evaluate the policies' effectiveness in the event of various scenarios such as the destruction of your office building, the inability of staff to commute to work, or the loss of connectivity to off-site data warehouses.

The skills and experience you bring to your job are equally valuable to your community. If you have any spare time, volunteer. As a QA professional and a concerned citizen, you can conduct research, analyze systems, and ask reasonable questions. Find an under-funded, under-staffed institution and offer your services. Learn about the emergency communications infrastructure, both the official government-affiliated media and the grassroots networks. Review evacuation plans. Inform yourself about public health facilities and procedures. Explain technology to the uninitiated. These activities can be especially rewarding for erstwhile dot-com employees who may be between jobs right now. Not only will you feel more useful when people start thanking you for your altruistic efforts; you may also impress someone enough to recommend you to your next employer.

In war, sports, and even office politics, some say that the best defense is a good offense. While aggressive action undoubtedly can chase evildoers back into their caves, and keep them there at least for a while, real security depends less upon armies winning battles than upon ordinary citizens remaining vigilant and looking out for each other in a spirit of trust and cooperation. Others may yearn to play Rambo, but most QA professionals are better suited for the role of Paul Revere. There will always be a worse case than the worst we can imagine; our goal should not be to envision cataclysms in minute detail but rather to ensure that the markets and institutions of civilized society can continue to function even when they are attacked. No matter where we work or live, we can still do a great deal to preserve and protect the quality of the American dream.

About the author

AgileConnection is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.