I'm trying to combine different aspects into simple recipes with make
.
Essentially, I am handling raw data and want to process it in a cross-product fashion.
I have input files
processing-scripts
├── all.awk
├── amd.awk
├── avg.awk
├── gm.awk
├── intel.awk
└── process.awk
processing-p256-mul
├── cycle.raw
├── mem.raw
├── total.raw
└── xmm.raw
Makefile
and my Makefile
GM_METRICS=cycle
AVG_METRICS=total xmm mem
METRICS=$(addsuffix .avg,$(AVG_METRICS)) $(addsuffix .gm,$(GM_METRICS))
ARCH=amd intel all
process-p256-mul: METHOD=p256-mul
process-p256-mul: $(addprefix processing-p256-mul/,$(foreach arch,$(addsuffix .,$(ARCH)), $(addprefix $(arch),$(METRICS))))
%/total.in: %/total.raw
sed -e 's@ \([0-9]*\)\r@:\1\r@' $(<) > $(@)
%/cycle.in: %/cycle.raw
sed -e 's@ *\([0-9]\+\).*(\(.*\))@\2:\1@' $(<) | grep -v Cycles > $(@)
%/xmm.in: %/xmm.raw
cp $(<) $(@)
%/mem.in: %/mem.raw
cp $(<) $(@)
amd.%.avg: %.in
awk -f ./processing-scripts/amd.awk -f ./processing-scripts/avg.awk -f ./processing-scripts/process.awk $(<) >$(@)
amd.%.gm: %.in
awk -f ./processing-scripts/amd.awk -f ./processing-scripts/gm.awk -f ./processing-scripts/process.awk $(<) >$(@)
intel.%.avg: %.in
awk -f ./processing-scripts/intel.awk -f ./processing-scripts/avg.awk -f ./processing-scripts/process.awk $(<) >$(@)
intel.%.gm: %.in
awk -f ./processing-scripts/intel.awk -f ./processing-scripts/gm.awk -f ./processing-scripts/process.awk $(<) >$(@)
all.%.avg: %.in
awk -f ./processing-scripts/all.awk -f ./processing-scripts/avg.awk -f ./processing-scripts/process.awk $(<) >$(@)
all.%.gm: %.in
awk -f ./processing-scripts/all.awk -f ./processing-scripts/gm.awk -f ./processing-scripts/process.awk $(<) >$(@)
Each different type of raw file, needs different preprocessing, total
needs this simple sed
expression, cycle
needs a bit more complicated one, xmm
and mem
only needs a simple copy. That works fine.
Now what I want to improve is the last part.
I need to generate (bash
-like expansion syntax) {intel,amd,all}.{{total,xmm,mem}.avg,cycle.gm}
.
(disregard for now everything below the process-p256-mul
rule)
Running make process-p256-mul -nk
gives me what I expect:
make: *** No rule to make target 'processing-p256-mul/amd.total.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/amd.xmm.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/amd.mem.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/amd.cycle.gm', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/intel.total.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/intel.xmm.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/intel.mem.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/intel.cycle.gm', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/all.total.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/all.xmm.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/all.mem.avg', needed by 'process-p256-mul'.
make: *** No rule to make target 'processing-p256-mul/all.cycle.gm', needed by 'process-p256-mul'.
Now adding the *in: *raw
recipes back in will generate the correct preprocessing rules. And adding the last six recipes in will also print the correct process commands.
My question is how do I write the last rules without being so verbose. Semantically:
%1.%2.%3: %1.%2.in
awk -f ./processing-scripts/%1.awk -f ./processing-scripts/%3.awk -f ./processing-scripts/process.awk $(<) >$(@)
I feel like I could get around the problem in the command with $(basename $<)
and $(basename $(basename $@))
but how would I make the target match to begin with?
PS: I'm also more than happy for advice on the foreach, if there is maybe a dedicated command for this... Feels a bit wrong. Or maybe point to a similar question, I'm struggling with finding the right keywords for my search. (As you can see in the title)
The following assumes GNU make
. There are 3 different problems to solve:
foreach-eval-call
and secondary expansion.avg
or gm
suffix; this is straightforward with foreach-eval-call
, a bit more tricky with secondary expansion.With foreach-eval-call
:
define MY_RULE
$1.%.$2: %.in
awk -f processing-scripts/$1.awk -f processing-scripts/$2.awk \
-f processing-scripts/process.awk $$< > $$@
endef
$(foreach a,$(ARCH),$(foreach b,avg gm,$(eval $(call MY_RULE,$a,$b))))
See this for the detailed explanation about foreach-eval-call
. Pay attention to the $$
in the macro definition they are essential.
With secondary expansion and make functions for the recipe:
TARGETS := $(foreach a,$(ARCH),$(addprefix processing-p256-mul/$a.,$(METRICS)))
process-p256-mul: $(TARGETS)
.SECONDEXPANSION:
$(TARGETS): $$(@D)/$$(word 2,$$(subst ., ,$$(@F))).in
awk -f processing-scripts/$(word 1,$(subst ., ,$(@F))).awk \
-f processing-scripts/$(word 3,$(subst ., ,$(@F))).awk \
-f processing-scripts/process.awk $< > $@
As suggested by John Bollinger you could also use shell substitutions to extract the architecture and the suffix in the recipe:
$(TARGETS): $$(@D)/$$(word 2,$$(subst ., ,$$(@F))).in
f=$(@F); a=$${f%%.*}; m=$${f##*.}; \
awk -f processing-scripts/$$a.awk -f processing-scripts/$$m.awk \
-f processing-scripts/process.awk $< > $@
See this for the detailed explanation about secondary expansion. Here too, pay attention to the $$
in the prerequisites list and in the recipe.