Descrambling Parallel Build Logs

[article]

SHELL:=$(shell pwd)/locker lockfile
all: a b c
a b c: ; @for n in 1 2 3 4 ; do echo $@-$$n && sleep 1 ; done
With this modification, the parallel build output might look like this:
b-1
b-2
b-3
b-4
a-1
a-2
a-3
a-4
c-1
c-2
c-3
c-4
Success! The output from individual commands is no longer interleaved with output from other commands. Not a bad return for the effort invested.

Shortcomings
Of course this solution is not perfect. As I said earlier, to really solve this problem correctly you need help from the make tool. The principal drawbacks of this approach are:

There is a small amount of additional overhead introduced by the addition  of the wrapper to the command. 

Output is handled on a per-command basis, not per-target. That means that  if you have a target with a multi-line rule body, it is possible that the  output of the different commands could get separated. 

The order of output is still non-deterministic. That means that if you run  the same build in parallel again, the build log might be in a different order.  For example:  

a-1
a-2
a-3
a-4
b-1
b-2
b-3
b-4
c-1
c-2
c-3
c-4

So it's not perfect, but I think it is demonstrably better than the previous sorry state of affairs. Here's hoping you find it useful. If you have other ideas for dealing with this problem, please share them in the comments!

About the author

Eric Melski's picture Eric Melski

Eric is Chief Architect for ElectricAccelerator, a high-performance implementation of make from Electric Cloud, Inc.  He obtained a BS in Computer Science from the University of Wisconsin in Madison in 1999.  In 2002 Eric co-founded Electric Cloud, where he has spent more than a decade developing distributed, parallel systems designed to accelerate build processes.  He is named on seven patents related to his work on build acceleration at Electric Cloud.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Oct 12
Oct 15
Nov 09
Nov 09