Review By: Harmon Avera, Jr.
07/02/2002This book starts out by presenting a rationale for evaluating software system architectures. Next, several evaluation methods are defined and discussed, with extensive step-by-step descriptions and checklists, and one or more actual case studies presented for each evaluation technique. Finally, it compares these various methods and discusses the buy-in and personnel issues involved in growing an Architecture Evaluation Capability within an organization.

The eleven chapters in the book are organized as follows:

- Chapters 1 and 2 provide a common starting point and terminology; when and why to evaluate an architecture; who’s involved; cost/benefits; and expected results.

- Chapters 3 through 8 define and discuss the evaluation methods and present case studies: 1) Architecture Tradeoff Analysis Method (ATAM): the major phases and steps, a short case study, fundamental concepts underlying the ATAM, and a detailed, in-depth application of the ATAM to a NASA system. 2) Software Architecture Analysis Method (SAAM): a specialized method used to evaluate architectures for modifiability and functionality. 3) Active Reviews for Intermediate Designs (ARID): used to test the feasibility and suitability of a set of services provided by a portion of an architecture.

- Chapters 9 through 11 compare and contrast the methods, explain how to start and grow evaluation capability within an organization, and present some conclusions and final thoughts.

Each chapter ends with a set of discussion questions and a section “For Further Reading.” There is a comprehensive list of references at the end of the book. The book is about 340 pages, including the “Preface” section and extensive index.

As with any testing, the reason to evaluate software architectures is to identify and mitigate risk. The authors’ thesis is that the wrong architecture will lead to project calamity, and this book presents evaluation methods designed to reduce the risk of choosing the “wrong” architecture. All of the evaluation techniques have some common steps:
1) Bring together representatives from all stakeholders (business drivers, management, design, and development);
2) Describe the architecture;
3) Identify the critical, business-driven quality attributes of the system;
4) Create scenarios used to map the quality attributes to architectural features; and
5) Evaluate the architectural approaches against these scenarios to identify risks, sensitivity points, and tradeoffs.
The differences between the methods are the “weight” (SAAM is more “lightweight” than ATAM) and architecture completeness (ARID can be used on partial or incomplete architectures).

This book, one in Addison-Wesley’s “SEI Series in Software Engineering,” is designed as a guidebook for practitioners of software architecture evaluation, written from the point of view of the evaluator. The Software Engineering Institute (SEI) is famous for developing the Capability Maturity Model (CMM) and I expected a dry, academic volume good for curing insomnia. Instead, I enjoyed reading it. The costs and benefits of doing the evaluations were identified. The steps for each method are clearly laid out with extensive checklists. The case studies show real-world problems as well as successes. The discussion of organizational and personnel issues associated with starting such evaluations was thoughtful and pertinent. I highly recommend this book for anyone involved in reducing software project risk.