Execute a Makefile in parallel except for some rules

I have a complex makefile with a lot of recipes. I would like run them with no parallel execution except for the generation of my objects files. I noticed that the .NOTPARALLEL target cannot take any prerequisites otherwise it would have been much easier to solve my issue.

My first guess was to use a nonexistent target named ".PARALLEL" with which I would have mentioned the objects files as dependancies like this:

SRC=$(wildcard *.c)
OBJ=$(SRC:.c=.o)           

.PARALLEL: $(OBJ)

%.o: %.c
    gcc –c –o$@ $< -M

a.out: $(OBJ)
    gcc –o$@ $^

A more functional solution I have found is to use an intermediate target. However, since MyObjects has no dependancies, make will always call MyObjects and recreate a.out.

%.o: %.c
    $(CC) –c –o$@ $< -M

MyObjects: 
    $(MAKE) -j $(OBJ)

a.out: MyObjects
    $(CC) –o$@ $(OBJ)

To avoid this I've found nothing better than using dummy files. I wrote this example to illustrate it:

NAMES = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

SRC = $(addsuffix .c, $(NAMES))
OBJ = $(patsubst %.c,%.o,$(SRC))
DUM = $(addsuffix .dummy,$(OBJ))

all: a.out

$(SRC):%.c: 
    touch $@

$(OBJ):%.o: %.c
    cp $< $@
    touch $(addsuffix .dummy, $@)

$(DUM):
    $(MAKE) -j8 $(OBJ)

a.out: $(DUM) $(OBJ)
    zip $@ $(OBJ)    

clean: 
    -rm *.o
    -rm *.out
    -rm *.c 
    -rm *.dummy

I'm sure this is not the best solution I can get. I would be glad to get some help.

P.S. MadScientist, thank you for your advices.

Solution

This is really not right:

MyObjects: $(OBJ)
        $(MAKE) -j $(OBJ)

This means that before make tries to build the MyObjects target, it will first try to update all the $(OBJ) files. Once that's all done, then it will try to build the MyObjects target by recursively invoking make to rebuild them again. Obviously that's not what you want. Plus you're using -j which is basically "infinitely parallel" and is likely (if you have enough object files) to bring your system to its knees.

You want something like this:

MyObjects:
        $(MAKE) -j5 $(OBJ)

As for your second question about trying to rebuild targets, there's no way we can help without some kind of specific example. Typically this happens because your rules are written incorrectly, and they don't actually update the target you told make they would. So for example, you have a target recipe_a but the rule for recipe_a updates some other target, not recipe_a.

I'll add a few notes based on your second question. Probably if you don't get it after this you should take this off of StackOverflow and ask on the [email protected] mailing list, or else consider breaking this up and asking several specific StackOverflow questions.

First, why you see make[1]: '15.o' is up to date. for every file in your recursive make: because make always prints that message for every target on the command line, so if you run make 1.o 2.o 3.o ... (doesn't matter whether you use -j or not or what value of -j you use) you'll get that message for every target which doesn't need to be rebuilt. Just as if you ran that same make command from the command line yourself.

Second, why you don't get a.out is up to date, because a.out is NOT up to date. It depends on the build target, and the file build doesn't exist, and thus it's out of date, and so it must be rebuilt every time. And that means anything that depends on the build target, like a.out, must be rebuilt every time. Which explains why it always re-runs the zip command.

Third, the behavior with all.c is because if you create a pattern rule like %.c: with no prerequisites, that tells make that it can create ANY file with a .c extension by running that command. Well, one of the targets you asked make to build is the all target. Since you didn't declare that as a .PHONY target, make tries to build it. Normally that attempt fails because make can't find any rules that know how to build all so nothing happens, but after you tell make how to build a .c file out of nothing (no prerequisites), then when make wants to build all it looks in its internal database of predefined rules and sees a pattern rule % : %.c, which tells make how to build an executable from a source file with the same name (on UNIX systems executables don't have any suffix like .exe: they're just make or cc etc.) So, make tries to run those rules and they fail.

For any target which you don't expect to actually be created, like all, clean, etc. you should declare them to be .PHONY so make won't try to build them.

As for your problem. I think the simplest thing to do is push the entire build of the zip file down into the recursive make, rather than trying to build the objects only in the recursive make. Something like this:

NAMES = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

SRC = $(addsuffix .c,$(NAMES))
OBJ = $(patsubst %.c,%.o,$(SRC))

all: recurse

recurse: non-parallel-targets
        $(MAKE) -j8 a.out PARALLEL=true

ifneq($(PARALLEL),true)
.NOTPARALLEL:
endif

%.o: %.c
        cp $< $@

a.out: $(OBJ)
        zip $@ $(OBJ)

init: $(SRC)

clean: 
        -rm *.o
        -rm *.out

.PHONY: all clean init