Learning from Concurrent, Parallel, and Distributed Systems Design

This month we do a bit of a context switch from the world of parallel development to the world of concurrent, parallel, and distributed systems design (and then back again). The purpose is to see if any of the same patterns of concurrent, parallel, and distributed processing apply to the case of concurrent, parallel, and distributed development.

The Multiple Dimensions of Parallel Development

Multiple projects, multiple variants, multiple products, multiple teams, multiple sites, multiple customer/install bases, … multi-everything! Why does it all have to be so complicated! Each "multi"-something introduces a new dimension of complexity and scale for software development. The more "multi's" we have, the more diverse and complex the task of managing, organizing, integrating, coordinating and tracking all of the work.

Agile development methods tend to focus on simplicity and keeping things simple. When it comes to process, they also believe that it’s usually better to start small and scale-up by adding incrementally instead of starting with a large all inclusive menu and trying to pare down. As a result, most agile methods don’t have too many explicit practices to handle the cases of multiple projects, multiple variants, multiple products, multiple teams, multiple sites, and multiple customer/install bases. Instead the preference is to first try to find ways to eliminate these scenarios before trying to find practices to handle them.

This is probably sound advice as many of us are often too quick to jump to a solution before asking if the problem is one that really should be solved rather than avoided. Still, for many of us, it is nonetheless the case that these scenarios are business realities that aren’t going to go away anytime soon. So we nevertheless must seek solutions for these problems.

Agile methods and the agile community sprang from software patterns and the patterns community. So patterns are often a good first place to look for finding such solutions. However, the existing body of patterns literature devoted to CM is substantially smaller than for software design patterns. When we cant find what we’re looking for in the CM patterns literature, we may need to look elsewhere.

Core Concepts for Concurrency
If we look at the world of concurrent/parallel and distributed systems design, there are many common concepts and solutions that may also apply to the domain of parallel development. We just need to comprehend them, and how to map them from their “native” domain into our own, so we can see if they are applicable to us!

The literature on concurrent/parallel and distributed computing is fraught with technical jargon about processors, processes, and threads (among other things). Here is an oversimplified primer of some basic concurrency concepts:

  • A process is a task for a processor to execute. It includes the logical flow of operations to execute, and an area in memory to store the results of computations. Processes may be heavyweight or lightweight; Lightweight processes are called threads.
  • A heavyweight process has its own separate flow of control/execution, and its own storage area (address space).
  • A lightweight process (or thread) has its own separate flow of control but executes in a shared address space with other threads

With that in mind, we can now describe concurrent processing, parallel processing, distributed processing, and multi-threading.

  • Concurrent processing amounts to doing more than one thing (executing more than one process) at the same time with the same processor.
  • Throw in another processor, and we have parallel processing: two processors executing at the same time to perform separate (but possibly related) tasks
  • Put them on different machines in a network, and we have distributed processing
  • Multi-threading (a.k.a. multi-threaded processing) is literally multi-tasking! It is when multiple threads are active at the same time for a single process.

For a better explanation of these and other concurrency concepts, I heartily recommend chapter 5 of 


About the author

Brad Appleton's picture Brad Appleton

Brad Appleton is a software CM/ALM solution architect and lean/agile development champion at a large telecommunications company. Currently he helps projects and teams adopt and apply lean/agile development and CM/ALM practices and tools. He is coauthor of the book Software Configuration Management Patterns, a columnist for the CMCrossroads and AgileConnection communities at Techwell.com,  and a former section editor for The C++ Report. You can read Brad's blog at blog.bradapp.net.

About the author

Steve Berczuk's picture Steve Berczuk

Steve Berczuk is a Principal Engineer and Scrum Master at Fitbit. The author of Software Configuration Management Patterns: Effective Teamwork, Practical Integration, he is a recognized expert in software configuration management and agile software development. Steve is passionate about helping teams work effectively to produce quality software. He has an M.S. in operations research from Stanford University and an S.B. in Electrical Engineering from MIT, and is a certified, practicing ScrumMaster. Contact Steve at steve@berczuk.com or visit berczuk.com and follow his blog at blog.berczuk.com.

About the author

Steve Konieczka's picture Steve Konieczka

Steve Konieczka is President and Chief Operating Officer of SCM Labs, a leading Software Configuration Management solutions provider. An IT consultant for fourteen years, Steve understands the challenges IT organizations face in change management. He has helped shape companies’ methodologies for creating and implementing effective SCM solutions for local and national clients. Steve is a member of Young Entrepreneurs Organization and serves on the board of the Association for Configuration and Data Management (ACDM). He holds a Bachelor of Science in Computer Information Systems from Colorado State University. You can reach Steve at steve@scmlabs.com.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!