PRE_TARGETDEPS fails when using parallel builds

I'm using a custom ressource management (in replacement to qrc) and I'm trying to integrate it to QtCreaor.

I have a Python script that generates a source file to be compiled. I use QMAKE_EXTRA_TARGETS/PRE_TARGETDEPS to tell QMake that this script must be executed before files are compiled. So I do, in my pro file:

CONFIG += ordered

generated_file.target   = my_custom_target
generated_file.commands = echo "Generating..." && d:/dev/vobs_ext_2015/tools_ext/python/Python34_light/python.exe $$PWD/pyc_res_generator.py -o $$PWD/generated/generated.cpp && echo "Generated!"
generated_file.depends  = FORCE

QMAKE_EXTRA_TARGETS            += generated_file

PRE_TARGETDEPS                 += my_custom_target

SOURCES                        += generated/generated.cpp

#DEPENDPATH = ./generated

With pyc_res_generator.py simply being:

#! /usr/bin/env python
# -*- coding: utf8 *-*

import argparse

parser = argparse.ArgumentParser(description="", formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument("-o", type=str, help="Output file name")

args = parser.parse_args()

with open(args.o, 'w') as output_file:
    output_file.write( "// This is valid C++!" )

To test this, I write some invalid C++ in generated.cpp (like fjkfkfk). When I compile (target is Android), I see in the log:

echo Generating... && python.exe pyc_res_generator.py -o generated/generated.cpp && echo Generated!
Generating... 
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ -c -Wno-psabi -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -DANDROID -Wa,--noexecstack -fno-builtin-memmove -std=c++11 -g -g -marm -O0 -fno-omit-frame-pointer -Wall -Wno-psabi -W -D_REENTRANT -fPIC -DQT_QML_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I..\TestQt -I. -IB:\QtCreator5_6_1\5.6\android_armv7\include -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtWidgets -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtGui -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtCore -I. -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\include -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\libs\armeabi-v7a\include -isystem B:\Android\android-ndk-r11b\platforms\android-9\arch-arm\usr\include -IB:\QtCreator5_6_1\5.6\android_armv7\mkspecs\android-g++ -o main.obj ..\TestQt\main.cpp
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ -c -Wno-psabi -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -DANDROID -Wa,--noexecstack -fno-builtin-memmove -std=c++11 -g -g -marm -O0 -fno-omit-frame-pointer -Wall -Wno-psabi -W -D_REENTRANT -fPIC -DQT_QML_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I..\TestQt -I. -IB:\QtCreator5_6_1\5.6\android_armv7\include -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtWidgets -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtGui -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtCore -I. -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\include -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\libs\armeabi-v7a\include -isystem B:\Android\android-ndk-r11b\platforms\android-9\arch-arm\usr\include -IB:\QtCreator5_6_1\5.6\android_armv7\mkspecs\android-g++ -o dialog.obj ..\TestQt\dialog.cpp
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ -c -Wno-psabi -march=armv7-a -mfloat-abi=softfp -mfpu=vfp -ffunction-sections -funwind-tables -fstack-protector -fno-short-enums -DANDROID -Wa,--noexecstack -fno-builtin-memmove -std=c++11 -g -g -marm -O0 -fno-omit-frame-pointer -Wall -Wno-psabi -W -D_REENTRANT -fPIC -DQT_QML_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I..\TestQt -I. -IB:\QtCreator5_6_1\5.6\android_armv7\include -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtWidgets -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtGui -IB:\QtCreator5_6_1\5.6\android_armv7\include\QtCore -I. -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\include -isystem B:\Android\android-ndk-r11b\sources\cxx-stl\gnu-libstdc++\4.9\libs\armeabi-v7a\include -isystem B:\Android\android-ndk-r11b\platforms\android-9\arch-arm\usr\include -IB:\QtCreator5_6_1\5.6\android_armv7\mkspecs\android-g++ -o generated.obj ..\TestQt\generated\generated.cpp
B:\QtCreator5_6_1\5.6\android_armv7\bin\moc.exe -DQT_QML_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -IB:/QtCreator5_6_1/5.6/android_armv7/mkspecs/android-g++ -IC:/Users/jp225611/Documents/TestQt -IB:/QtCreator5_6_1/5.6/android_armv7/include -IB:/QtCreator5_6_1/5.6/android_armv7/include/QtWidgets -IB:/QtCreator5_6_1/5.6/android_armv7/include/QtGui -IB:/QtCreator5_6_1/5.6/android_armv7/include/QtCore -I. -IB:\Android\android-ndk-r11b/sources/cxx-stl/gnu-libstdc++/4.9/include -IB:\Android\android-ndk-r11b/sources/cxx-stl/gnu-libstdc++/4.9/libs/armeabi-v7a/include -IB:\Android\android-ndk-r11b/platforms/android-9/arch-arm//usr/include ..\TestQt\dialog.h -o moc_dialog.cpp
..\TestQt\generated\generated.cpp:1:1: error: 'fjkfkfk' does not name a type
   // This is valid C++!
   ^
makefile:670: recipe for target 'generated.obj' failed
mingw32-make: *** [generated.obj] Error 1
mingw32-make: *** Waiting for unfinished jobs....
Generated!
13:59:08: Le processus "B:\QtCreator5_6_1\Tools\mingw492_32\bin\mingw32-make.exe" s'est terminé avec le code 2.
Erreur lors de la compilation/déploiement du projet TestQt (kit : android_armeabi-v7a)
When executing step "Make"
13:59:08: Temps écoulé : 00:03.

And I see that generated\generated.cpp was correctly generated by pyc_res_generator.py (it now contains // This is valid C++!)...but apparently too lately as the compiler saw fjkfkfk... As you can see, the output reports:

Generating...
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ .... ..\TestQt\generated\generated.cpp
Generated...

While one would expect:

Generating...
Generated...
B:\Android\android-ndk-r11b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-g++ .... ..\TestQt\generated\generated.cpp

This only occurs when parallel builds are enabled (I set MAKEFLAGS environment variable to '-j8`)

It looks like my custom command generated_file is started before generated\generated.cpp is compiled, but the system does not wait for it to end before actually compiling generated\generated.cpp.

Am I doing something wrong?

Solution

PRE_TARGETDEPS (custom targets) is not the good approach here as generated_file.target must match exactly the Makefile target name which may be vary on different options:

shadow build enabled or not
host platform (using slashs or backslash as a folder separator)
probably targetted compiler/platform

However, custom compiler apparently works much better. That's how moccing files is done. A custom compiler can generate a new file to be compiled (and then the system waits for the file to be generated before compiling it, even when performing parallel builds).

# Set fuiles to be generated in a variable
TO_GENERATE = $$PWD/generated/generated.cpp

# Specify custom command output file:
custom_generator.output  = $$PWD/generated/generated.cpp
# Specify custom command:
custom_generator.commands = 'echo "Generating..." && python $$PWD/pyc_res_generator.py -o $$PWD/generated/generated.cpp && echo "Generated..."'
# dependency:
custom_generator.depends = FORCE
# link to input file variable
custom_generator.input = TO_GENERATE
# link to variable to store generated file to
custom_generator.variable_out = SOURCES
# add to qmake:
QMAKE_EXTRA_COMPILERS += custom_generator

Then, you don't even need to specify SOURCES += $$PWD/generated/generated.cpp. And also, $$PWD/generated/generated.cpp gets deleted upon clean!

This works perfectly as expected!