The Trouble with $(wildcard)

[article]

$(warning Preexisting file: $(WILDCARD_LIST))

.PHONY: all
all: $(DIRECTORY)b.foo
@echo $(WILDCARD_LIST)
@echo $(LS_LIST)

$(DIRECTORY)b.foo:
@touch $@

And here's the output:

Makefile:6: Preexisting file: wildcard returned 'subdir/a.foo'
wildcard returned 'subdir/a.foo'
ls returned 'subdir/a.foo subdir/b.foo'

Notice now that make appears to be behaving like the first example above, the subdir/b.foo file that was made by the Makefile is invisible to $(wildcard) and doesn't appear even though it was created and ls found it.

Why This Happens

The unexpected, and apparently inconsistent, results above are because GNU Make contains its own cache of directory entries. $(wildcard) reads from that cache (and not directly from disk like ls) to get its results. When that cache is filled is vital to understanding the results the $(wildcard) will return.

GNU Make only fills the cache when it has too (for example, when it needs to read the directory entries to satisfy a $(wildcard) or other globbing request). If you know that GNU Make only fills the cache when needed then it's possible to explain the results seen above.

In the first example, GNU Make fills the cache for the current working directory when it starts. Hence the file b.foo doesn't appear in the output of $(wildcard) because it wasn't present when the cache was filled. In the second example, GNU Make hasn't filled the cache with entries from subdir until they were needed. The entries were first needed for the $(wildcard) which is performed after subdir/b.foo is created and hence subdir/b.foo does appear in the $(wildcard) output. In the final example, the $(warning) happens at the start of the Makefile which filled the cache (because it did a $(wildcard)) and hence subdir/b.foo was missing from the output of $(wildcard) for the duration of that make.

Predicting when the cache will be filled is very hard. $(wildcard) will fill the cache, but so will use of a globbing operator like * in the target or prerequisite list of a rule. Consider this example Makefile that builds two files (subdir/b.foo and subdir/c.foo) and does a couple of $(wildcard)'s:

DIRECTORY=subdir/

.PHONY: all
all: $(DIRECTORY)b.foo
@echo $(WILDCARD_LIST)
@echo $(LS_LIST)

$(DIRECTORY)b.foo: $(DIRECTORY)c.foo
@touch $@
@echo $(WILDCARD_LIST)
@echo $(LS_LIST)

$(DIRECTORY)c.foo:
@touch $@

The output may surprise you:

wildcard returned 'subdir/a.foo subdir/c.foo'
ls returned 'subdir/a.foo subdir/c.foo'
wildcard returned 'subdir/a.foo subdir/c.foo'
ls returned 'subdir/a.foo subdir/b.foo subdir/c.foo'

Even though the first $(wildcard) is being done in the rule that makes subdir/b.foo and after the touch that created subdir/b.foo there's no mention of subdir/b.foo in the output. Nor is there mention of subdir/b.foo in the output of the ls.

That's because the complete block of commands is expanded into its final form before any of the lines in the rule are run. So the $(wildcard) and $(shell ls) are done before the touch has run.

The output of $(wildcard) is even more unpredictable if the make is run in parallel with the -j switch. In that case the exact order in which the rules are run is not predictable and hence the output of $(wildcard) can be even less predictable.

Recommendations

Don't use $(wildcard) in a rule; only use $(wildcard) in the Makefile at parsing time (i.e. before any rules start running). If you restrict use of $(wildcard) to parsing time you can be assured of consistent results: $(wildcard) will show the state of the filesystem before make was run.

About the author

John Graham-Cumming's picture John Graham-Cumming

John Graham-Cumming is Co-Founder at Electric Cloud, Inc . Prior to joining Electric Cloud, John was a Venture Consultant with Accel Partners, VP of Internet Technology at Interwoven, Inc. (IWOV), VP of Engineering at Scriptics Corporation (acquired by Interwoven), and Chief Architect at Optimal Networks, Inc. John holds BA and MA degrees in Mathematics and Computation and a Doctorate in Computer Security from Oxford University. John is the creator of the highly acclaimed open source POPFile project. He also holds two patents in network analysis and has others pending.

AgileConnection is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

May 04
May 04
May 04
Jun 01