Search code examples
pythoncsvmakefile

Multiple targets from one script, copying to multiple different directories in Makefile

Scenario

I am working on updating a script that generates source code files for both C and Java at build time, for each of our builds.
This needs to run on both Windows and Linux.
Two file types need to be created for each language: Enums and Tables.

Each set of four files (C Enum, C Table, Java Enum, Java Table) is generated from a corresponding CSV file.
Given an input CSV file called A.csv, the output files are AEnum.inc, ATable.inc, AEnum.java, and ATable.java.

All four files can be created at once by calling a python script, with an activated venv, like so:

python main.py --input A.csv --output ./output --enum-c --table-c --enum-java --table-java

This is shortened in the makefile with a function called run_autogen_all.

Once the files have been created, they need to be copied to different directories:

  • C enums & tables go in $(C_AUTOGEN)/
  • Java enums go in $(JAVA_ENUM)/
  • Java tables go in $(JAVA_TABLE)/

Problem

I have not been able to figure out how to ensure the python script runs only once (runtime is non-negligible), while also copying all of the files to the correct directories.

How can I achieve this behavior with Make?

Attempted solution

Here is my current solution that works, but it runs the python script twice and copies files twice.

MAIN_4_CSV_FILES = A.CSV B.CSV C.CSV D.CSV E.CSV F.CSV G.CSV H.CSV I.CSV J.CSV K.CSV L.CSV M.CSV N.CSV O.CSV P.CSV Q.CSV R.CSV S.CSV T.CSV U.CSV V.CSV W.CSV X.CSV Y.CSV Z.CSV
.PHONY : JavaFiles CFiles
JavaFiles : *.java
CFiles : *.inc

*.java *.inc : $(MAIN_4_CSV_FILES)
    $(call run_autogen_all,$^)
    $(COPYF) *Enum.java $(JAVA_ENUM)
    $(COPYF) *Table.java $(JAVA_TABLE)
    $(COPYF) *.inc $(C_AUTOGEN)

Running make JavaFiles CFiles -n gives the following output

python "../../../../../build/Common/CSV_Parser/autogenerate_from_csv.py" --output "./" --csv-files A.CSV B.CSV C.CSV D.CSV E.CSV F.CSV G.CSV H.CSV I.CSV J.CSV K.CSV L.CSV M.CSV N.CSV O.CSV P.CSV Q.CSV R.CSV S.CSV T.CSV U.CSV V.CSV W.CSV X.CSV Y.CSV Z.CSV
cp -f *Enum.java ../../../../Project/java/enums
cp -f *Table.java ../../../../Project/java/tables
cp -f *Enum.inc ../../../../Project/c
cp -f *Table.inc ../../../../Project/c
python "../../../../../build/Common/CSV_Parser/autogenerate_from_csv.py" --output "./" --csv-files A.CSV B.CSV C.CSV D.CSV E.CSV F.CSV G.CSV H.CSV I.CSV J.CSV K.CSV L.CSV M.CSV N.CSV O.CSV P.CSV Q.CSV R.CSV S.CSV T.CSV U.CSV V.CSV W.CSV X.CSV Y.CSV Z.CSV
cp -f *Enum.java ../../../../Project/java/enums
cp -f *Table.java ../../../../Project/java/tables
cp -f *Enum.inc ../../../../Project/c
cp -f *Table.inc ../../../../Project/c

Previous attempts

I've tried many options, here are a few that didn't work

%Enum.inc %Enum.java %Table.inc %Table.java :
    $(call run_autogen_all,$<)
    $(call copy_attrib,$(notdir $@),$@)

AFiles = $(C_AUTOGEN)/AEnum.c $(C_AUTOGEN)/ATable.c $(JAVA_ENUM)/AEnum.java $(JAVA_TABLE)/ATable.java
$(AFiles) : A.csv
[...]

JavaEnums : $(JAVA_ENUM)/AEnum.java $(JAVA_ENUM)/BEnum.java [...]

$(JAVA_ENUM)/%Enum.java : %Enum.java
    $(call run_autogen_all,$*.csv)
    $(call copy_attrib,$(notdir $@),$@)
[...]

%Enum.java %Enum.c %Table.java %Table.c : %.csv
    $(call run_autogen_all,$<)
    $(call copy_attrib,$(notdir $@),$@)
[...]

I also tried messing with .INTERMEDIATE and .SECONDARY to no avail.

Solution

  • Assuming you use GNU make and all generated files are generated in the current directory you can use a pattern rule with 4 targets.

    MAIN_4_CSV_FILES := $(wildcard *.CSV)
TARGETS          := $(patsubst %.CSV,$(C_AUTOGEN)/%Enum.inc,$(MAIN_4_CSV_FILES))
TARGETS          += $(patsubst %.CSV,$(C_AUTOGEN)/%Table.inc,$(MAIN_4_CSV_FILES))
TARGETS          += $(patsubst %.CSV,$(JAVA_ENUM)/%Enum.java,$(MAIN_4_CSV_FILES))
TARGETS          += $(patsubst %.CSV,$(JAVA_TABLE)/%Table.java,$(MAIN_4_CSV_FILES))

.PHONY: all
all: $(TARGETS)

$(C_AUTOGEN)/%Enum.inc $(C_AUTOGEN)/%Table.inc $(JAVA_ENUM)/%Enum.java $(JAVA_TABLE)/%Table.java: %.CSV
   python main.py --input $< --output ./output --enum-c --table-c --enum-java --table-java
   mv $*Enum.inc $*Table.inc $(C_AUTOGEN)
   mv $*Enum.java $(JAVA_ENUM)
   mv $*Table.java $(JAVA_TABLE)

    GNU make considers the 4 targets as grouped and runs the recipe only once to build them. Don't use $@ in the recipes of such rules (at least not before you read the documentation).

    Once you will have tested this try maybe to re-introduce your macros.