Makefile Optimization: $(shell) and := go Together

[article]

expanded it remains that way unless it is redefined using the = operator. This means that if you append to a simply expanded macro the text being appended is expanded before the append.

For example,

FOO=foo
 BAR=bar
 BAZ=baz
 FOOBAR := $(FOO) $(BAR)
 FOOBAR += $(BAZ)
 BAZ=bazzy 

results in FOOBAR being foo bar baz. If = had been used instead of := then when $(BAZ) was appended it would not have been expanded and the resulting FOOBAR would have been foo baz bazzy.
Take a look at this example Makefile.

CWD = $(shell pwd)
 SRC_DIR=$(CWD)/src/
 OBJ_DIR=$(CWD)/obj/
 OBJS = $(OBJ_DIR)foo.o $(OBJ_DIR)bar.o $(OBJ_DIR)baz.o
 $(OBJ_DIR)%.o: $(SRC_DIR)%.c ; @echo Make $@ from $<
 all: $(OBJS)
 	@echo $? $(OBJS)
  

It gets the current working directory into CWD, defines a source and object directory as sub-directories of the CWD, defines a set of objects (foo.o, bar.o and baz.o) to be built in the OBJ_DIR, sets up a pattern rule showing how to build a .o from a .c and finally states that by default the Makefile should build all the objects and print out a list of those that were out of date ($? is the list of prerequisites of a rule that were out of date) and a full list of objects.

You might be surprised to learn that this Makefile ends up making eight shell invocations just to get the CWD value. Imagine how many times GNU Make would make costly calls to the shell in a real Makefile with hundreds or thousands of objects!

There are so many calls to $(shell) because the Makefile uses recursively expanded macros: i.e. macros whose value is determined when the macro is used and not at definition time. OBJS references OBJ_DIR three times which references CWD each time; every time OBJS is referenced there are three calls to $(shell pwd). Any other reference to SRC_DIR or OBJ_DIR (e.g. the pattern rule definition) results in another $(shell pwd).

But there's a quick fix for this. Just change the definition of CWD to simply expanded by inserting a : to turn = into :=. Since the working directory doesn't change during the Make we can safely get it once:

CWD := $(shell pwd) 

Now there's a single call out to the shell to get the working directory. In a real Makefile this could be a large time saver.

Since it can be hard to follow through a Makefile to see everywhere a macro is used there's a simple trick that will cause Make to print out the exact line at which a macro is expanded. If we insert $(warning Call to shell) in the definition of CWD so that its definition is

CWD = $(warning Call to shell)$(shell pwd) 

then we get the following output when we run Make:

Makefile:8: Call to shell
 Makefile:8: Call to shell
 Makefile:10: Call to shell
 Makefile:10: Call to shell
 Makefile:10: Call to shell
 Make /somedir/obj/foo.o from /somedir/src/foo.c
 Make /somedir/obj/bar.o from /somedir/src/bar.c
 Make /somedir/obj/baz.o from /somedir/src/baz.c
 Makefile:11: Call to shell
 Makefile:11: Call to shell
 Makefile:11: Call to shell
 /somedir/obj/foo.o /somedir/obj/bar.o /somedir/obj/baz.o /somedir/obj/foo.o
 /somedir/obj/bar.o /somedir/obj/baz.o
  

The $(warning) doesn't change the value of CWD, but it does output a message to STDERR. From the output you can see that eight calls to the shell and which lines in the Makefile caused them.

If CWD is defined using := the $(warning) trick verifies that CWD is expanded only once:

Makefile:1: Call to shell
 Make /somedir/obj/foo.o from /somedir/src/foo.c
 Make /somedir/obj/bar.o from /somedir/src/bar.c
 Make /somedir/obj/baz.o from /somedir/src/baz.c
 /somedir/obj/foo.o /somedir/obj/bar.o /somedir/obj/baz.o /somedir/obj/foo.o
 /somedir/obj/bar.o /somedir/obj/baz.o
  

Quick Recipe
A quick way to find out if you are using the expensive combination of =

About the author

John Graham-Cumming's picture John Graham-Cumming

John Graham-Cumming is Co-Founder at Electric Cloud, Inc . Prior to joining Electric Cloud, John was a Venture Consultant with Accel Partners, VP of Internet Technology at Interwoven, Inc. (IWOV), VP of Engineering at Scriptics Corporation (acquired by Interwoven), and Chief Architect at Optimal Networks, Inc. John holds BA and MA degrees in Mathematics and Computation and a Doctorate in Computer Security from Oxford University. John is the creator of the highly acclaimed open source POPFile project. He also holds two patents in network analysis and has others pending.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!