The Past
Allocating an independent testing team to test a software product is often a controversial issue, mainly because development managers are frequently not interested in paying for these services due to budget and time constraints. On the other hand, senior management generally is interested in these services-senior managers know the importance of testing and how the tests protect the organization and their products from embarrassing failures. 

One of the most common excuses from the development managers is that independent testing is too expensive and lengthy-the project schedule has to fit the customer's schedule. Thus, there is no time left for independent testing activities. The organization I work for faces this constantly. As a systems test manager, I have been asked how I can reduce the timeline for testing a software product. I am faced with this dilemma largely because the projects developed at my organization have changed a lot recently. In our first years, we were asked to develop commercial, off-the-shelf software (COTS). Adding systems testing to a COTS project was not difficult and no one questioned it. All groups involved in developing the project were interested in testing. The reason was that the project schedules were long (eighteen to twenty-four months), they had many (six to ten) engineers working on them, and several artifacts were produced during development time.

Under those conditions, we applied a complete set of extensive tests and techniques, following our testing methodology fully. We worked together and provided the development teams with suggestions and comments for the projects during the test cycles. There were no customers to specify details for the COTS products. The testers were simply another team with experience and another perspective.

The Present
Then the Internet arrived. Everything changed, including the customers, and several tools have become available to help the development teams build powerful solutions faster and cheaper.

My organization has followed the wave. Nowadays, we have been developing Internet solutions with Documentum, BEA, and FileNet Panagon tools for telecom, chemical, and energy supplier companies, locally and offshore. We have also added CMM Level 3 to our skills. It has been one of the greatest challenges for the organization to add the new technologies and skills since 1998.

Everything has had a large impact on the way we run testing activities. The effort and time for developing the products have dropped dramatically. The development team size was reduced to three to four engineers. The effort usually ranges from eight to twenty personal months (PMs), and the project length is shorter, from three to six months. Now, some new questions have arisen to join the old ones:

• How can the customer pay 50 percent of the project costs for qualifying a software product?
• How can a systems test be added to a project in this new scenario?

The new questions presented new challenges for the systems test group. Functioning independently from the rest of the organization, we had to define how we would work in this increasingly cost-sensitive scenario and not be pressured.

Our task was to propose a new approach to our testing methodology to the organization. The proposal would guide us in defining how the systems test group should work on future projects. In addition, product quality would not suffer through reduced testing costs, and the project delivery date would be anticipated (the famous QCD!).

The Resource Allocation Process
First we properly defined the new scenario and problems we faced. We noticed that the differences among the projects were related to the effort, not to the technology or to any other project characteristic. And the major problem was estimating the testing team size and the testing project length.

We decided that the projects under test would be classified by effort. Then, we created the following classifications:

• Projects up to seven PMs: The testing team would apply the "Free Allocation" system.
• Projects from eight to twelve PMs: The testing team would apply the "Intermediate Allocation" system.
• Projects more than twelve PMs: The testing team would apply the "Full Allocation" system.

Based on this classification, we wrote the "Resource Allocation Process" containing a new approach to our testing methodology. The systems test group believes that the questions will be answered when applying the Resource Allocation Process to projects. That means cost-sensitive projects still can benefit from an independent testing team involved throughout the project lifecycle.

The process was presented to all groups (development, sustaining, and documentation) at my organization and the development managers decided it should be applied in the next projects.

The Process in Detail

Free Allocation System

This allocating system is for small projects in which the development team allocates engineers up to seven PMs to build the product. In this situation, the testing team provides one test engineer to the development team. The test engineer will be managed completely by the development leader, but will use his experience to make sure testing is conducted thoroughly.

The test engineer may work in an ad hoc manner. He might not follow our
processes or templates to run the tests. And the testing methodology may be put
aside occasionally during the tests (unless you acknowledge these ad hoc
activities as built into, or part of, the larger methodology).

The development team is not obliged to request a test engineer. Also, we are not obliged to provide one when we are requested. It depends on the availability of testing staff. Of course, one of my interests is to increase the number of test engineers allocated to a project. We are always working to improve the ratio of testers to developers.

Before we developed this allocation system, the development team rarely asked for testing on this kind of project, due to the testing costs that would represent more than 50 percent of the total project costs. As the costs will not be an issue, it is expected that the requests for testing will become more frequent.

Of course, allocating test engineers to development in an ad hoc manner isn't good for them or for the organization. The Project Management Institute has warned about this. So, we established some conditions for limiting it.

Conditions
The first condition is limiting the days when the test engineer will be available to the development team. We established twenty-one continuous working days (one month). One PM will be charged to the project budget.

If it is necessary to allocate the test engineer extra time, then he may be assigned for up to ten working days, depending on availability. In this situation, the extra time of the test engineer and a test leader will be charged.

In this system, as the projects are completely run by the development teams without testing's participation, development is responsible for allocating any hardware or software resources for performing the testing activities. The systems test group provides only the engineer workstation and personal software.

In order to track what is being accomplished by the testing staff in the project, the test engineer sends a weekly report by mail to a test leader, reporting the activities accomplished, some metrics (yet to be defined), and the number of days spent. The metrics will help the systems test group and development managers determine if the tests were effective.

The test engineer will inform the development manager about the product quality during the test cycle.

Intermediate Allocation System
This system is for projects that have eight to twelve PMs allocated. The development manager is obliged to ask for testing services if the project fits this criterion.

The main purpose of this allocation system is to provide a light testing service to the development team, not to fully qualify the software. In this system, the development manager is the main person responsible for product quality.

As these sorts of projects are not big and we have already implemented CMM Level 3, we have decided to take more risks. Taking more risks means that applying the new approach in the projects would make our testing methodology less expensive and more flexible.

Some nonessential activities were suppressed from our testing methodology, and they are not executed at all in this allocation system. Some others were merely adjusted. The intention is to allow the project to be delivered earlier and for less cost. For that reason, some risks may become real.

The following activities have been changed in the testing methodology for this allocating system:

• have only one regression phase
• do not have a time gap between the execution and regression phases (time for fixing defects)
• report only defects with severity from one to three during the execution phase
• do not give suggestions for product improvement during the testing cycle
• test the solution only in one environment (even if there are others)
• use simplified document templates for documenting the testing activities
• do not verify the user's manual

But the main change is that the testing schedule is fixed for all projects in this allocating system. The preparation phase has ten working days with two test engineers at 100 percent allocation. The execution and regression phases have eleven working days with one test engineer at 100 percent allocation. Then, the testing cycle is twenty-one working days altogether. One team leader is also allocated throughout the period, but only at 50 percent allocation. The total testing resource allocation represents two PMs.

As the effort is always two PMs, the testing costs are also the same for all projects that fit this criterion. Therefore, the development manager may know previously how long the testing services would take and how much they would cost. Predictability is going to be a major advantage with this new approach.

It is easy to notice that the testing costs do not seriously impact the project costs in this allocation system. The testing costs vary from 15 to 19 percent of the total project costs. The testing costs were around 40 percent of the total costs when the testing methodology was applied fully in this kind of project.

Full Allocation System
This allocation system is for projects to which the development team has allocated more than twelve PMs. The development manager is obliged to ask us for testing services if the project fits this criterion.

In this case, the systems test group applies our testing methodology fully. The testing resources are estimated and allocated in the project based on the software requirements. This is described in my article "The Test Estimation Process."

The tests are planned, executed extensively, and reported by an independent testing team as predicted in our testing methodology, using testing tools when required. Systems test has the main responsibility for diagnosing the product quality at the end of the test cycle. The development manager follows our advice regarding release of the product to the field. The testing costs for these projects vary from 20 to 30 percent of the total project costs.

Final Consideration
This approach is not for all organizations and projects. I could apply it to mine because I know the development teams at my organization. Also, CMM Level 3 allows us to handle more risks without affecting our output quality; however, when testing products from development groups outside our organization, we would not take these risks. You need to know who is providing you with the artifacts to be tested in your organization. Then you will be able to plan and allocate your testing resources in an efficient manner.