In so many of the development projects I've been on, the requirements were difficult to pinpoint at the outset. The business side knew generally what they wanted, but the specifics-things that could affect the fundamental design-could not be anticipated. Datamart and data warehousing applications have been particularly sensitive to this inability to fully lay out the particulars of reports that will come in the future. Web applications woven into complex business agreements are equally pliable. In programming these projects, I've often drawn on lessons I learned flying for a commuter airline on those stormy days where most people would rather be at work than traveling.

In the world of air flight, a flight plan rigidly defines the checkpoints, route, time, and fuel required to get from point A to point B. Using atmospheric data plugged into aircraft performance equations, accurate predictive estimates of navigation are obtained. Once in flight, the actual progress is compared to the plan. In a world of pure physics, the flight plan estimates of fuel and time exactly equal the actual flight. When the weather conditions are unpleasant-such as fog, thunderstorms, or frozen precipitation-the actual flight events deviate from the plan.

Take a typical summer flight from New York to Washington, D.C., with thunderstorms along the route. Thunderstorms, obstacles that are inflexible and too big to go around, often pop up with short notice and mushroom in size. Ever heard of that description on your projects? Before departure, a flight plan is prepared defining the route with fuel and time estimates. The pilots review the real-time weather radar and weather reports to get a general impression of storm movement and intensity, developing a practical avoidance strategy. Due to the constraints of airspace usage, combined with the dynamics of thunderstorm development and movement, the strategy is not expressed as a full plan but rather an overview of prospective options.

The flight crew departs with the big picture of the weather. In flight, they use their aircraft weather radar for ongoing real-time pictures of the storms. The aircraft weather radar provides the most practical data for the flight, because it provides the only picture that matters: the position of the storms relative to the aircraft. The flight crew interprets their airborne weather radar display in a continuous feedback loop to reach conclusions about the safest route. The flight crew deviates from the flight plan and forges a new route based on conditions that are, by their nature, impossible to predict. The flight plan still serves the key purpose as a baseline for comparing the anticipated to the actual, to verify that the progress being made is acceptable for the amount of fuel on board.

In the world of software, there are those projects where the weather causes low visibility. The overall flight plan is just as important, but the exact route to be followed cannot be defined due to changing conditions. In these cases, the exact route can only be determined by getting started and reacting to conditions through observation and measurement. In fact, certain conditions cannot be known until you are under way. If you delay in an effort to acquire requirements that are so absolute that the path is completely defined, you will never depart.

Extreme Programming, also known as Agile Development, tries to get the flight off the ground and get the software project to its destination successfully. The "Extreme" approach is to take off, keeping your options open, then making adjustments along the way.

I once worked on a Web-based intranet datamart for a large homecare provider. We had to collect lots of data from five different legacy systems describing nursing visits to patients. Core elements included collecting the data from the systems, modeling it, normalizing it, and then generating reports off that data. The business group could only imagine the tip of the iceberg of the kinds of reports they could get. Ideas about how to validate this operational data against hard accounting data were just emerging. So we departed with a broad plan that tackled the fundamentals: data acquisition and normalization. We then built some rudimentary reports. This approach helped us get high enough that we could see over the horizon. And I suppose like during a scary flight, we saw a lot of storms.

Measurement is critical to Extreme Programming. You must measure your progress closely because it is the only way to be sure you are on track. With each measurement, you correct. Start with small, achievable components that can be verified with unit tests as soon as you create them. If your project is like a flight of 400 miles, consider points that are 25 miles apart and make those small segments a trip in and of themselves. Once you are 25 miles down the road, make sure all is safe with the sponsors, testers, and other project participants, then do the next 25 miles.

In the datamart project I described, the first part we focused on was data collection and normalization. Once that was pretty stable, we proceeded with our first reports on the Web. Next, we came up with validation methods that tied back to the accounting system-which helped us refine our normalizations. With better data validity, we were able to add more detailed drilldowns and different dimensions to the reports that could only be fully conceptualized once the data elements were refined.

Have courage. You might go along the way and see an insurmountable wall of thunderstorms ahead of you. In fact, they may be closing in around you. Don't get cornered. Turn around. Get yourself out the way you came in. Regroup. Get some information from external resources that help paint a more complete picture. With the new information considered, get right back on with it.

When we first started the datamart project, we totally overloaded our SQL 6.5 database. The entire load routine was defective, and it was not broken into small enough units. In effect, what we were developing was a consistent method to crash SQL, and we soon became experts at rebooting. So we regrouped. We rewrote part of the normalization as a simple, text-based, pre-cleaning program. We then broke the one database into five-to more logically model the five feeder systems that were in place-and achieved reliable and dramatically faster load times.

Software projects cover a lot of ground. The secret lies in small, achievable goals-each one providing safe refuge and alternatives if the unexpected occurs. The continual iterations gradually fit together to cover the distance from departure to destination. The ground gets traversed and a path is carved through the many obstacles. That 400-mile trip may take 475 miles of ground to cover, but that is what conditions dictated. The goal should not be to cover a 400-mile trip in 400 miles. The goal is to get to the destination in the minimum time that is consistent with the reality of the presented landscape.