Rebuilding When CPPFLAGS Changes

Summary:

GNU Make has no way of detecting that some targets ought to be rebuilt, because it doesn't take into account changing the commands. If, for example DEBUG=1 causes the flags passed to the compiler to change then the target ought to be rebuilt. This article shows how, in a few lines of GNU Make code, to make that happen.

For example, what happens if you do a non-debug build (by typing make) and then run a debug build by typing make DEBUG=1. Unless the build has been structured so that the names of targets are dependent on whether the build is debug or non-debug, nothing happens at all.

This article shows how, in a few lines of GNU Make code, to make that happen.

An Example

Here's an example Makefile that's used throughout this article to demonstrate the rebuilding when commands change system. To make the operation of the system very clear I've avoided using built-in GNU Make rules so this Makefile isn't as simple as it could be.

The Makefile creates two .o files: foo.o and bar.o by compiling corresponding .c files. The compilation is done using the built-in variable COMPILE.C (which will be something like normally be the name of a suitable compiler for your system, references to variables like CPPFLAGS and use of $@ and $< to compile the right thing).

There's a specific reference to $(DEBUG), it's turned into a preprocessor variable called DEBUG using the compiler's -D option. I haven't bothered showing the contents of foo.c and bar.c as they are irrelevant.

all: foo.o bar.o

foo.o: foo.c
$(COMPILE.C) -DDEBUG=$(DEBUG) -o $@ $<

bar.o: bar.c
$(COMPILE.C) -o $@ $<

Here's what happens if we first run make (which means that DEBUG is undefined) followed by make DEBUG=1.

$ make
g++ -c -DDEBUG= -o foo.o foo.c
g++ -c -o bar.o bar.c

Now foo.o and bar.o have been created (with DEBUG empty) and so typing make again does nothing:

$ make
make: Nothing to be done for `all'.

Typing make DEBUG=1 also does nothing, even though the object file foo.o would likely be different if it were rebuilt with DEBUG defined.

$ make DEBUG=1
make: Nothing to be done for `all'.

The signature system described below will correct that, with very little work for the Makefile maintainer.

The Example Revisited

To fix the problem described above this article introduces a helper Makefile called signature. The contents and working of signature are described below, but first let's look at how the example Makefile is modified to used the signature helper.

include signature

all: foo.o bar.o

foo.o: foo.c
$(call do,$$(COMPILE.C) -DDEBUG=$$(DEBUG) -o $$@ $$<)

bar.o: bar.c
$(call do,$$(COMPILE.C) -o $$@ $$<)

-include foo.sig bar.sig

Three changes have been made to the file: firstly include signature has been added at the start so that the code that handles the signature updating is included. The commands in the two rules have been wrapped with $(call do,...) and the $ signs for each command have been quoted with a second $.

Lastly for each .o file being managed by signature there's an include of a corresponding .sig file. The final line of the Makefile includes foo.sig (for foo.o) and bar.sig (for bar.o). Notice that -include is used in case the .sig file is missing.

Before seeing how this works here are some example of it in operation. First run a clean build (i.e. with no .o files present) and then rerun make to see that there's nothing to do:

$ make
g++ -c -DDEBUG= -o foo.o foo.c
g++ -c -o bar.o bar.c
$ make
make: Nothing to be done for `all'.

But now setting DEBUG to 1 on the make command-line cause foo.o to rebuild because its 'signature' (i.e. the actual commands to be run to build foo.o have changed.

$ make DEBUG=1
g++ -c -DDEBUG=1 -o foo.o foo.c

Of course, bar.o was not rebuilt because it was truly up to date (it's object was new and there were no command changes). Running make DEBUG=1 again says that there's nothing to be done, but just typing make rebuilds foo.o again because DEBUG is now undefined.

$ make DEBUG=1
make: Nothing to be done for `all'.
$ make
g++ -c -DDEBUG= -o foo.o foo.c

The signature system also works for variables that are hidden within a recursive variable. In GNU Make COMPILE.C actual expands CPPFLAGS to create the complete compiler command-line. Here's what happens if we modify CPPFLAGS on the command-line by adding a definition:

$ make CPPFLAGS+=-DFOO=foo
g++ -DFOO=foo -c -DDEBUG= -o foo.o foo.c
g++ -DFOO=foo -c -o bar.o bar.c

Both foo.o and bar.o were rebuilt because CPPFLAGS had changed (and because CPPFLAGS was part of the commands used to build those two object files).

Of course, changing a variable that isn't referenced doesn't cause anything to be updated. Here's an example starting from a clean build and then redefining SOMEVAR.

$ make
g++ -c -DDEBUG= -o foo.o foo.c
g++ -c -o bar.o bar.c
$ make SOMEVAR=42
make: Nothing to be done for `all'.

How Signature Works

To understand how this works the first place to look is inside a .sig file. The .sig files are automatically generated by signature for each rule that uses the $(call do,...) (the details of how are later on).

Here, for example, is the contents of the foo.sig file after the first clean build was run:

$(eval @ := foo.o)
$(eval % := )
$(eval < := foo.c)
$(eval ? := foo.force)
$(eval ^ := foo.c foo.force)
$(eval + := foo.c foo.force)
$(eval * := foo)

foo.o: foo.force

$(if $(call sne,$(COMPILE.C) -DDEBUG=$(DEBUG) -o $@ $<,g++ -c -DDEBUG= -o foo.o foo.c),$(shell touch foo.force))

The first seven lines capture the state of the automatic variables as defined when the foo.o rule is being processed. These are needed so that the current commands for a rule can be compared with the commands the last time the rule was run.

Next comes the line foo.o: foo.force. This says that foo.o must be rebuilt is foo.force is newer. It's this line that causes foo.o to get rebuilt when the commands change, and it's the next line that touches foo.force if the commands have changed.

The long $(if ...) statement uses the GMSL (see http://gmsl.sf.net/) sne (string not equal) to compare the current commands for foo.o (by expanding them) against their value the last time they were expanded. If the commands have changed then $(shell touch foo.force) is called.

Since the .sig files are processed when the Makefile is being parsed (they are just Makefile's themselves read using include), all the .force files will have been updated before any rules run. And so this small .sig file does all the work of forcing an object file to rebuild when the commands change.

The .sig files themselves are created by signature:

include gmsl

last_target :=

dump_var = \$$(eval $1 := $($1))

define new_rule
@echo "$(call map,dump_var,@ % < ? ^ + *)" > $S
@$(if $(wildcard $F),,touch $F)
@echo $@: $F >> $S
endef

define do
$(eval S := $*.sig)$(eval F := $*.force)$(eval C := $1)
$(if $(call sne,$@,$(last_target)),$(call new_rule),$(eval last_target := $@))
@echo "$(subst $$,\$$,$$(if $$(call sne,$1,$C),$$(shell touch $F)))" >> $S
$C
endef

signature include the GNU Make Standard Library and then defines the important do macro used to wrap the commands in a rule. When do is called it creates the appropriate .sig file containing the state of all the automatic variables.

The capture of automatic variables is done by the new_rule macro called by do which uses the GMSL map function to call another macro (dump_var) for each of @ % < ? ^ + *. new_rule also ensures that the corresponding .force file has been created.

Lastly, do writes out the complex $(if ...) statement that contains both the unexpanded and expanded versions of the commands for the current rule. And then it actually runs the commands (that's the $C) at the end.

Limitations

There are a small number of limitations to this technique. Firstly if the commands in a rule contain any side effects (e.g. they call $(shell ...) then the system may misbehave if there was an assumption that the $(shell ...) was only called once.

Secondly, it's vital that signature is included before any of the .sig files.

And lastly, if the Makefile is edited and the commands in a rule change the signature system will not notice. If you do that then it's vital to regenerate the corresponding target so that the .sig is updated.

Conclusion

Not a lot of code, for a lot of benefit. Let me know if you use it in a large project!

Lastly, reader Semih Cemiloglu wrote to say that a while back I'd promised to write an article on tracing which rules in a Makefile fire and in what order. Although some of that is covered in recent articles on the GNU Make Debugger, I'll return in April with the missing piece! Thanks for the reminder Semih.

User Comments

1 comment

Hi,

I am looking for a way to have the makefile have the command generating my target as a dependence of my target.

The signature process you describe is close to that. I have face the following issues :

- if the command line contains commas, linker command with gcc do have comma for compiler options, the call to sne will have more that 2 arguments and fails every time.

- Support of target-specific variable values

Since 2006, is there any new way of doing this ?

To solve the comma issue, I recorded the full command in a file and read the file as the 2nd argument of sne.

Regards,
Stan

About the author

John Graham-Cumming

John Graham-Cumming is Co-Founder at Electric Cloud, Inc . Prior to joining Electric Cloud, John was a Venture Consultant with Accel Partners, VP of Internet Technology at Interwoven, Inc. (IWOV), VP of Engineering at Scriptics Corporation (acquired by Interwoven), and Chief Architect at Optimal Networks, Inc. John holds BA and MA degrees in Mathematics and Computation and a Doctorate in Computer Security from Oxford University. John is the creator of the highly acclaimed open source POPFile project. He also holds two patents in network analysis and has others pending.

User Comments

About the author

John Graham-Cumming

Connect with Me