How can I force gnu make to not build recipe in parallel?

How can I tell gnu make not to build some recipe in parallel. Let's say I have the following makefile :

sources = a.xxx b.xxx c.xxx
target  = program

all : $(target)

$(target) : $(patsubst %.xxx,%.o,$(sources))
    $(CXX) -o $@ $<

%.o : %.cpp
    $(CXX) -c -o $@ $<

%.cpp : %.xxx
    my-pre-pro开发者_运维技巧cessor -o $@ $<

However, the my-pre-processor command create temporary files with fixed name (I cannot change this). This is working fine if I just use make without the -j parameter. However, if the -j option is used, the build sometimes fails because two concurrent invocation of my-pre-processor overwrite their temporary files.

I'd like to know if there is a way to tell make that it must not build the try to parallelize the execution of the %.cpp : %.xxx recipes.

Solution 1

GNU Make contains the special built-in pseudo-target .NOTPARALLEL

Example:

.PHONY: all clean

.NOTPARALLEL:

anotherTarget: dependency1

Solution 2

You can also use the -j <n>,--jobs[=<n>] flag on the command line where n is the number of recipies allowed to run in parallel.

Usage: make -j <n> or make --jobs=<n>

Example: make -j 1 or make --jobs=1

note: omitting <n> will allow an arbitrary number of recipes to be executed, only limited by your system's available resources

Solution 3

Finally, you can assign the command line flag in solution 2 to the MAKEFLAGS variable from within your Makefile

Example: MAKEFLAGS := -j 1 or MAKEFLAGS := --jobs=1

A related solution is to specify the exact order you want things built (rather than saying, "don't build in parallel").

To specify the exact order, you can use order-only prerequisites. Here's GNU make's man page on it:

Occasionally, however, you have a situation where you want to impose a specific ordering on the rules to be invoked without forcing the target to be updated if one of those rules is executed. In that case, you want to define order-only prerequisites. Order-only prerequisites can be specified by placing a pipe symbol (|) in the prerequisites list: any prerequisites to the left of the pipe symbol are normal; any prerequisites to the right are order-only:

targets : normal-prerequisites | order-only-prerequisites

And here's the example they offer:

OBJDIR := objdir
OBJS := $(addprefix $(OBJDIR)/,foo.o bar.o baz.o)

$(OBJDIR)/%.o : %.c
        $(COMPILE.c) $(OUTPUT_OPTION) $<

all: $(OBJS)

$(OBJS): | $(OBJDIR)

$(OBJDIR):
        mkdir $(OBJDIR)

I had to use order only prerequisites on a make dist rule due to a race condition. The dist recipe depended on a distclean and diff rules, and the diff rule performed an svn diff -r XXX to show the exact changes. On occasion, make would delete the diff it just created because the clean rule would run after the diff rule.

This is a horrible kludge, but it will do the job:

b.cpp: a.cpp

c.cpp: b.cpp

Or if there are actually a lot of these, you can have a few stiff drinks and do this:

c-sources = $(sources:.xxx=.cpp)

ALLBUTFIRST = $(filter-out $(firstword $(c-sources)), $(c-sources))
ALLBUTLAST = $(filter-out $(lastword $(c-sources)), $(c-sources))
PAIRS = $(join $(ALLBUTLAST),$(addprefix :,$(ALLBUTFIRST)))

$(foreach pair,$(PAIRS),$(eval $(pair)))

(This works in GNUMake, I don't know about other versions.)

If the temporary files are created in the current working directory, you may be able to use subdirectories (not pretty, but rare):

sources = a.xxx b.xxx c.xxx
target  = program

all : $(target)

$(target) : $(patsubst %.xxx,%.o,$(sources))
    $(CXX) -o $@ $<

%.o : %.cpp
    $(CXX) -c -o $@ $<

%.cpp : %.xxx
    mkdir $@.d
    s=`realpath $<` && cd $@.d && my-pre-processor -o ../$@ "$${s}" || { $(RM) -r $@.d && false; }
    $(RM) -r $@.d

Also, since you are using syntax but not features that are exclusively available to GNU make, please note that the following equivalent Makefile should be more portable

sources = a.xxx b.xxx c.xxx
target  = program

all : $(target)

$(target) : $(sources:.xxx=.o)
    $(CXX) -o $@ $<

.cpp.o:
    $(CXX) -c -o $@ $<

.xxx.cpp:
    mkdir $@.d
    s=`realpath $<` && cd $@.d && my-pre-processor -o ../$@ "$${s}" || { $(RM) -r $@.d && false; }
    $(RM) -r $@.d

.PHONY: all
.SUFFIXES: .xxx .cpp .o

Also note that GNU make's intrinsic .cpp.o: rule allows for users to specify flags on the command line, (similar to)

.cpp.o:
    $(CXX) $(CPPFLAGS) $(CXXFLAGS) -c -o $@ $<

which your users may like when they need to provide, say, custom include directories via -L...

You would do well to study the operation of the ylwrap tool that comes with automake: It solves most of the same problems for old versions of lex and yacc:

http://git.savannah.gnu.org/cgit/automake.git/tree/lib/ylwrap

On linux you can use flock file -c "command": https://man7.org/linux/man-pages/man1/flock.1.html. I have solved a problem with GPU resource synchronization using the flock.

###############################
%.wav: %.txt
    flock .cuda.lock -c "python synthesize.py --in $< --out $@ --dev cuda:0"
################################
%.mp3: %.wav
    ffmpeg -i $< $@
################################

If I call: make 1.mp3 2.mp3 3.mp3 -j3 - it will sync python, but not ffmpeg.

I run into the same problem (my sw signing service did not accept multiple concurrent requests). My solution was to add a mutex in the recepie:

%.cpp : %.xxx
    @ until mkdir /tmp/make.lock 2>/dev/null ; do sleep 2 ; done
    my-pre-processor -o $@ $<
    @ rmdir /tmp/make.lock

This allows all other parts of the makefile to run in parallel, but there will only be one "my-pre-processor" running at the time.

@airenas's suggestion to use flock can be generalized as a utility Makefile you can include and use as in this test makefile:

Contents of NOTPARALLEL_test.mk

include NOTPARALLEL.mk

.ONESHELL:
$(shell mkdir -p t/NOTPARALLEL)

.NOTPARALLEL+=t/NOTPARALLEL/%
t/NOTPARALLEL/%::
    date > $@
    cat $@
    sleep 2

t: t/NOTPARALLEL/1 t/NOTPARALLEL/2 t/NOTPARALLEL/3

${USE_NOTPARALLEL}

which depends upon the included utility:

contents of NOTPARALLEL.mk

define _flockCmdScript :=
#!/bin/env bash
flock "$${1}" -c "$${2}"
endef

_flockCmdPath:=$(dir $(lastword $(MAKEFILE_LIST)))_flockcmd
.SECONDARY: ${_flockCmdPath}
${_flockCmdPath}: private SHELL=/bin/bash
.PHONY: flockcmd

PPID!=echo $PPID
_flock_prefix!=mktemp $${TMPDIR-/tmp}/.flock_XXX_NOTPARALLEL.$(notdir $(firstword $(MAKEFILE_LIST)))
override define NOTPARALLEL 
$(shell flock ${_flock_prefix}_make_flockCmdPath -c $$'if [ ! -e "${_flockCmdPath}" ] ; then cat > ${_flockCmdPath} <<< \'${_flockCmdScript}\' ; fi ;')
$1: private SHELL=${_flockCmdPath}
$1: private .SHELLFLAGS=${_flock_prefix}_$(subst /,.,$1)
endef

## define macro te be exanded at bottom of makefiles which `include
## NOTPARALLEL.mk` with the line: `${USE_NOTPARALLEL}'
USE_NOTPARALLEL=$(eval $(foreach pattern,${.NOTPARALLEL},$(call NOTPARALLEL,${pattern})))

## distributed & installed with this utility is NOTPARALLEL_test.mk
## which instructively may be called as:
t:
    $(MAKE) -O -j10 -B -f NOTPARALLEL_test.mk # which contains `include NOTPARALLEL.mk`
    $(MAKE) -O -j10 -B -f NOTPARALLEL_test.mk .NOTPARALLEL=t/% -B -j10 # all recipes targetting t/% are evaluated sequentially
    $(MAKE) -O -j10 -B -f NOTPARALLEL_test.mk NOTPARALLEL= -B -j10 # all recipes are run in parallel (since macro NOTPARALLEL (note: no leading period) is defined as empty).

You can then invoke multiple different usages of NOTPARALLEL_test file like this:

make -O -j10 -B -f NOTPARALLEL.mk  t

Here's some documentation:

PURPOSE: include NOTPARALLEL.mk in your makefiles when you'd like to use Make's parallel execution (e.g. -j/--jobs) but want to disable it for selected pattern rules. It defines two variables .NOTPARALLEL and USE_NOTPARALLEL. Add (e.g. using Makes += operator) to .NOTPARALLEL those pattern rules whose matching targets ought never have their recipes executed in parallel with each other (though still allowed to be executed in parallel with any other target in the makefile). You can also provide it at the command line. Put ${USE_NOTPARALLEL} at the end of your makefile to activate it.

It works to subvert GNUMake's --jobs feature by changing Make's SHELL and .SHELLFLAGS so the recipes run as an argument to flock. creating a lock file allowing only one instance of the recipe to run at a time.

Since flock fails with NFSv4 (and for performance in general), the lockfile is stored in $[TMPDIR} (defaulting to /tmp) (which is assumed to not be NFS mounted). The lock file is named after the current makefile, the process ID of the current Makefile, and the pattern itself.

The implementation override the SHELL variable to the bash script _flockcmd which in turn calls flock $1 -c $2. This script is created on demand in the same directory as NOTPARALLEL.mk when first needed.

This approach will only work when the current SHELL provides flock (as do sh & bash); specificially it is NOT generally expected to work in "advanced" uses of make in which the makefile has changed SHELL (e.g. to perl, or to sqlite, or to R). This is because the recipe is passed to, and executed by, flock as a shell command. Allowing for alternate SHELLs might be implemented in the future.

Note: it is hard to imagine the value of this whole approach without using .ONESHELL, but it in NOT required.

NB: recipes holding a lock will still use up one of Make's process slots.