Put a bunch of developers in an office with a mission to produce a specific software product and it’s only a matter of time before the product starts appearing. Now throw in a new requirement that dictates that development will be done by three or four teams distributed across different cities, countries, or time zones, and the results are no longer so predictable. How are the different teams going to work together to get the job done? Isn’t it better just to put everyone together in the same office?

There are many reasons that organizations or government agencies use distributed development. Maybe it’s a NATO project where multiple countries have been politically mandated to develop the software. Perhaps different subcontractors have different areas of expertise that need to come together. Maybe a series of mergers has left a particular software team spread out over several locations.

Whatever the reason, a software development team, and indeed the entire product team, needs to do all it can to ensure that development proceeds smoothly.

In any event, distributed development will require more effort on your part to encourage team members to communicate. Because the nature of distributed development is that teams are spread out, it’s likely that less communication will take place. To handle this problem you need to remove the barriers to communication. This can be done, for example, by supporting on-line code reviews—stored in the CM repository—or through on-line meetings for design reviews, change control boards, problem review boards, etc.

As your team members can no longer walk down the corridor to chat with peers and find out who is doing what, it’s critical that your CM repository and tools provide a clear communication of the project picture – priorities, assignments and status. The repository or tool that you use must collect data better (automatically when possible) and it must present the information more readily and in a pro-active way. This is to ensure that team members, otherwise unwilling to take the effort to get information, don’t proceed with their job with less information than they might use in a non-distributed environment.

For proper global development, there are three CM repository scenarios I'd like to discuss: distributed data, centralized sites, and synchronized multiple sites.

Divide and Conquer – Distributed Data
The general approach to using distributed development is to divide and conquer. In this approach you  spread the functionality along logical boundaries, such as are naturally defined by devices, system layers or subsystems. This reduces inter-site development conflicts. 

Every so often you bring the teams together with all of the software from each of the sites, so that an effort can be made to integrate them together. This has some merit, but also a few drawbacks.

For one thing, you must ensure that infrequent integration does not cause long delays while you work out the integration issues. This strategy flies in the face of continuous integration practices.

The integration will take some time, since it is done infrequently – so there will be more problems to diagnose and solve.  By the time a successful, consistent integration is attained, development will have moved on and integration work will have to be merged back into each site.

Another issue with this approach is that it takes a considerable effort to partition the software among the various sites.  This is not just an initial partitioning problem, but an ongoing problem as the need for new software components arises, and as the teams and expertise bases shift over time. The strategy of “divide and conquer” works best when it’s only the deliverables (i.e. not the source code) that have to come together for separate devices or subsystems.  But even then, we’re splitting our expertise into three or four compartments rather than taking advantage of  architectural expertise across all sites.

Centralized Site
The next alternative is to use a centralized operation where all of the software resides in one site. Typically, integration builds, including nightly builds, are performed at this centralized site, with the results being distributed nightly to the other sites.

A necessary concession in such a solution is to mirror the central site to all of the other sites so that local developer and integration builds can be performed quickly. This allows for delta operations, code searches, and other methods to be performed without the latency of operating within a long distance network. Some advantages of using a centralized solution include:

• Access to a consistent set of CM data through a central repository/site ensures that queries across the project are simple and current.
• There is no need to administer the partitioning and re-synchronization of project data.
• There is a central point for simpler, consistent repository backups.

The cross-project queries are more current and generally much faster than in the “divide and conquer” scenario.Centralized repositories permit an easy cloud computing option, which has a number of advantages, including:

• Team morale is kept at a constant because nobody has a preferred site advantage. 
• Cloud solutions typically have mirroring capabilities options.
• Integration builds that are performed in the cloud are readily accessible at all sites.

However, even in the cloud, there will be response time delays typical of any remote operation.  Mirrored local sites are more responsive for complex queries, such as workspace synchronization. For small projects with hundreds of files, mirroring is not essential, but for larger ones, the delays add up.

As well, there will be costs associated with using cloud products that will not be insignificant; this includes a certain level of access administration. Whether or not these costs are offset by reduced operational costs depends on the nature of administration for your non-hosted solution.

However, network interruptions can still be a problem for remote users. Additionally, check-in operations involving numerous large files will necessarily incur longer (upload) delays for all but those at the central site. This in turn can lead to less frequent developer check-ins.

Synchronized Multiple Sites
A third alternative is to use synchronized multiple sites. In this scenario, each site is a complete CM repository of its own, with little or no dependence on other sites. A single master process, somewhere on the network, orders all transactions and typically assigns each a unique sequence number. The sites cooperate in distributing the files of the transactions to all sites.

Although there are still inherent upload delays between sites, users are able to communicate efficiently with their local repository while the onus is on the server to sort out transaction ordering and to deal with upload delays across the network.

While this may seem similar to the centralized operation scenario, it has unique advantages. For one thing, clients (users) are able to receive response times typical of the central site at all sites. Integration teams can work at each site independently if desirable; this is often the case if time zones are far apart.

Network outages are also far less likely to cause delays, even for transactions and rarely during query operations.

Users can move from one site to another without changing how they operate, since the entire repository is up-to-date and operates identically at each site. As such, separate training is not required based on location, as is the case in a centralized scenario.

Finally, there are multiple on-line backups of the repository, which reduces the need for local backups and allows for rapid disaster recovery. Ideally, if there were a local CM server disk crash, users would link to a different site and continue working, perhaps with the delays observed by a remote user.

Still, there are caveats. If we depend on each site to act independently on its repository, we must ensure that the CM solution behaves identically at each site. This may mean using the exact same environment and platform at each site or it may mean using tools that will guarantee platform and environment independence; I prefer the latter so as not to restrict the IT organization at each site.

I recommend using repositories that are easily audited for synchronization. What happens if a CM software bug manifests itself on one platform but not another, or in one release but not another? It might be fair to require all sites to run the same version of the CM software, but an operating system condition (e.g. “disk full”) or a human error can cause a difference in operation. I would feel better if I could know whether or not the repositories are all in sync. If they are not, recovery must be fast and simple.

Ideally, we want the tool to look and feel like it runs on a centralized repository while at the same time reaping the benefits of synchronous multiple site operation.

Distributed ALM
Distributed development doesn’t end with the best “CM” (i.e. source code control) solution.  It’s one thing to support global development from a source code perspective, but quite another from a product perspective. The entire application life-cycle has to be managed.

Sales managers around the world need to understand the differences between the new release and what’s at their customer’s site.  Developers need to have current priority lists, especially in an agile environment. Technical writers need to peruse the product repository to ensure documentation they write accurately reflects the state of the product.

So how do some common tools deal with global development? Well to start with, centralized tools will just extend the tool set from centralized SCM to centralized ALM.

CollabNet provides layered centralized ALM component products for Subversion. As centralized solutions, typically in the cloud, there is no issue here. Similarly, Perforce runs primarily as a centralized solution and has a few companion products that have a similar centralized architecture. Serena’s Dimensions is a centralized solution for its entire ALM offering.

IBM’s ClearCase, a distributed data solution, is a source code repository, but it’s companion product, ClearQuest, supports many of the other ALM functions. Each has it’s own multiple-site solution and they work together, though with a fair bit of glue.

Neuma’s CM+ extends its synchronous multiple site operation through all ALM functions. In this case, there’s only one capability that has to be maintained and monitored, and all ALM data is always globally available. CM+ also includes the ability to selectively exclude classes of data from specific sites, as might be required for security reasons.

GIT, primarily a distributed data solution, is not easily expanded to support ALM solutions, at least not with the same architecture. Typically, ALM is handled through centralized third-party tools which complement the distributed GIT operation. This is not necessarily bad, but GIT’s architecture makes it more difficult to grow into a full ALM solution and even beyond source code control into a full CM solution.

Going Global?
There are obviously good or bad choices you can make when selecting technology based on a global development perspective. But what if you have the choice to do or not do distributed development. What if you’re the decision maker?

My recommendation is simply to avoid it if you can. A tightly knit development team operating in a single location will have a synergy of its own. As soon as you introduce a second site, there is a natural tendency for an “us” versus “them” mentality to take a hold of team members whenever an issue arises. Yes, this can happen locally as well, but it is easier to deal with when you can regularly put all of the people involved in the same room together.

You can’t always avoid global development.  Perhaps your team may eventually outgrow the building and split into two locations. Whatever the reason, do your best to understand the technical issues that will help you succeed, because the non-technical issues will be challenging enough.