Search code examples
makefile

Gnumake cross product target matching


I'm trying to combine different aspects into simple recipes with make. Essentially, I am handling raw data and want to process it in a cross-product fashion.

I have input files

processing-scripts
├── all.awk
├── amd.awk
├── avg.awk
├── gm.awk
├── intel.awk
└── process.awk
processing-p256-mul
├── cycle.raw
├── mem.raw
├── total.raw
└── xmm.raw
Makefile

and my Makefile

GM_METRICS=cycle
AVG_METRICS=total xmm mem
METRICS=$(addsuffix .avg,$(AVG_METRICS)) $(addsuffix .gm,$(GM_METRICS))
ARCH=amd intel all
process-p256-mul: METHOD=p256-mul
process-p256-mul: $(addprefix processing-p256-mul/,$(foreach arch,$(addsuffix .,$(ARCH)), $(addprefix $(arch),$(METRICS))))

%/total.in: %/total.raw
    sed -e 's@ \([0-9]*\)\r@:\1\r@' $(<) > $(@)
%/cycle.in: %/cycle.raw
    sed -e 's@   *\([0-9]\+\).*(\(.*\))@\2:\1@' $(<) | grep -v Cycles > $(@)
%/xmm.in: %/xmm.raw
    cp $(<) $(@)
%/mem.in: %/mem.raw
    cp $(<) $(@)

amd.%.avg: %.in
    awk -f ./processing-scripts/amd.awk   -f ./processing-scripts/avg.awk -f ./processing-scripts/process.awk $(<) >$(@)
amd.%.gm: %.in
    awk -f ./processing-scripts/amd.awk   -f ./processing-scripts/gm.awk  -f ./processing-scripts/process.awk $(<) >$(@)
intel.%.avg: %.in
    awk -f ./processing-scripts/intel.awk -f ./processing-scripts/avg.awk -f ./processing-scripts/process.awk $(<) >$(@)
intel.%.gm: %.in
    awk -f ./processing-scripts/intel.awk -f ./processing-scripts/gm.awk  -f ./processing-scripts/process.awk $(<) >$(@)
all.%.avg: %.in
    awk -f ./processing-scripts/all.awk   -f ./processing-scripts/avg.awk -f ./processing-scripts/process.awk $(<) >$(@)
all.%.gm: %.in
    awk -f ./processing-scripts/all.awk   -f ./processing-scripts/gm.awk  -f ./processing-scripts/process.awk $(<) >$(@)

Each different type of raw file, needs different preprocessing, total needs this simple sed expression, cycle needs a bit more complicated one, xmm and mem only needs a simple copy. That works fine. Now what I want to improve is the last part.

I need to generate (bash-like expansion syntax) {intel,amd,all}.{{total,xmm,mem}.avg,cycle.gm}.

(disregard for now everything below the process-p256-mul rule) Running make process-p256-mul -nk gives me what I expect:

make: *** No rule to make target 'processing-p256-mul/amd.total.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/amd.xmm.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/amd.mem.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/amd.cycle.gm', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/intel.total.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/intel.xmm.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/intel.mem.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/intel.cycle.gm', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/all.total.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/all.xmm.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/all.mem.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/all.cycle.gm', needed by 'process-p256-mul'.

Now adding the *in: *raw recipes back in will generate the correct preprocessing rules. And adding the last six recipes in will also print the correct process commands.

My question is how do I write the last rules without being so verbose. Semantically:

%1.%2.%3: %1.%2.in
     awk -f ./processing-scripts/%1.awk   -f ./processing-scripts/%3.awk -f ./processing-scripts/process.awk $(<) >$(@)

I feel like I could get around the problem in the command with $(basename $<) and $(basename $(basename $@)) but how would I make the target match to begin with?

PS: I'm also more than happy for advice on the foreach, if there is maybe a dedicated command for this... Feels a bit wrong. Or maybe point to a similar question, I'm struggling with finding the right keywords for my search. (As you can see in the title)


Solution

  • The following assumes GNU make. There are 3 different problems to solve:

    • Compute the list of targets; you apparently know how to do this with the make functions, so let's skip this (but see below a slightly simpler solution than yours).
    • Associate targets and prerequisites; this is the tricky part. There are basically two options: foreach-eval-call and secondary expansion.
    • Write the recipes; here also there are several options to extract the architecture name and the avg or gm suffix; this is straightforward with foreach-eval-call, a bit more tricky with secondary expansion.

    With foreach-eval-call:

    define MY_RULE
    $1.%.$2: %.in
         awk -f processing-scripts/$1.awk -f processing-scripts/$2.awk \
           -f processing-scripts/process.awk $$< > $$@
    endef
    $(foreach a,$(ARCH),$(foreach b,avg gm,$(eval $(call MY_RULE,$a,$b))))
    

    See this for the detailed explanation about foreach-eval-call. Pay attention to the $$ in the macro definition they are essential.

    With secondary expansion and make functions for the recipe:

    TARGETS := $(foreach a,$(ARCH),$(addprefix processing-p256-mul/$a.,$(METRICS)))
    process-p256-mul: $(TARGETS)
    
    .SECONDEXPANSION:
    
    $(TARGETS): $$(@D)/$$(word 2,$$(subst ., ,$$(@F))).in
        awk -f processing-scripts/$(word 1,$(subst ., ,$(@F))).awk \
          -f processing-scripts/$(word 3,$(subst ., ,$(@F))).awk \
          -f processing-scripts/process.awk $< > $@
    

    As suggested by John Bollinger you could also use shell substitutions to extract the architecture and the suffix in the recipe:

    $(TARGETS): $$(@D)/$$(word 2,$$(subst ., ,$$(@F))).in
        f=$(@F); a=$${f%%.*}; m=$${f##*.}; \
        awk -f processing-scripts/$$a.awk -f processing-scripts/$$m.awk \
          -f processing-scripts/process.awk $< > $@
    

    See this for the detailed explanation about secondary expansion. Here too, pay attention to the $$ in the prerequisites list and in the recipe.