What is the proper way of using a source generator in CMake

In my C++ project I'm using a source generator to embed some resources into the binary.

I use CMake to build my project and my code works but had some issues. I am pretty sure that what I want to accomplish is possible but I didn't find any answer online.

The current problems I have are:

The generator runs every time, even if the input files did not change. This is not too big of a deal because it is really fast, but I hopped there was a better way to do it
While using Ninja the generator runs at every build (as described above) without rebuilding every time. I think that Ninja sees that the file has not changed and does not build it again, but when I make changes in the resources change it still uses the old version. It takes another build to "realize" that the generated file has changed and rebuild it
While using Make the code rebuilds every time, even when the generated file does not change, resulting in wasted build time

In both cases (looking at the output) the generator runs before the compiler.

This situation is not unsustainable but I was wondering if a better solution was possible. Here's a code snippet from my CMakeLists.txt

add_subdirectory(Generator)
file(GLOB RESOURCES Resources/*)
add_custom_command(
    OUTPUT src/Resources/Generated.hpp src/Resources/Generated.cpp
    COMMAND Generator ${RESOURCES}
    DEPENDS ${RESOURCES}
    DEPENDS Generator
)

add_custom_target(Generated DEPENDS src/Resources/Generated.hpp src/Resources/Generated.cpp)
add_dependencies(${PROJECT_NAME} Generated)

Here's a minimal reproducible example, sorry for not having it before.

EDIT: I implemented the functional solution from the correct answer in the fix branch, on the same repo. It might be useful for future reference :)

Solution

This one is interesting, because there are multiple errors and stylistic issues, which partially overlap each other.

First off:

file(GLOB_RECURSE SRC src/*.cpp src/*.hpp)
add_executable(${PROJECT_NAME} ${SRC})

While convenient in the beginning, globbing your sources is not a good idea. At some point you will have a testme.cpp in there that should not be built with the rest, or a conditionally_compiled.cpp that should only be compiled if a certain option is set. You end up compiling sources that you really did not intended to.

In this case, the file src/Generated.hpp from your git repository. That file is supposed to be generated, not checked out from repo. How did it even get in there?

add_custom_command(
    OUTPUT src/Generated.hpp
    COMMAND ${PROJECT_SOURCE_DIR}/generator.sh
            ${PROJECT_SOURCE_DIR}/Resources/data.txt
            > ${PROJECT_SOURCE_DIR}/src/Generated.hpp
    DEPENDS Resources/data.txt
    DEPENDS ${PROJECT_SOURCE_DIR}/generator.sh
)

Do you see the output redirection there? You wrote to ${PROJECT_SOURCE_DIR}. That is not a good idea. Your source tree should never have anything compiled or generated in it. Because these things end up being committed with the rest... like it happened to you.

Next issue:

add_custom_target(Generated DEPENDS src/Generated.hpp)

This creates a make target Generated. Try it: make Generated. You keep getting the following output:

[100%] Generating src/Generated.hpp
[100%] Built target Generated

Obviously it does not realize that Generated.hpp is already up-to-date. Why not?

Let's look at your custom command again:

add_custom_command(
    OUTPUT src/Generated.hpp
    COMMAND ${PROJECT_SOURCE_DIR}/generator.sh
            ${PROJECT_SOURCE_DIR}/Resources/data.txt
            > ${PROJECT_SOURCE_DIR}/src/Generated.hpp
    DEPENDS Resources/data.txt
    DEPENDS ${PROJECT_SOURCE_DIR}/generator.sh
)

What if I told you that your OUTPUT is never actually generated?

Quoting from CMake docs on add_custom_command, emphasis mine:

OUTPUT

Specify the output files the command is expected to produce. If an output name is a relative path it will be interpreted relative to the build tree directory corresponding to the current source directory.

So your output claims to be to the binary tree, but your command's redirection is to the source tree... no wonder the generator keeps getting re-run.

Having your header generated in the wrong location does not give you a compiler error, because that header from your source tree gets picked up by your GLOB_RECURSE. As that one keeps getting re-generated, your executable keeps getting recompiled as well.

Try this from your build directory:

mkdir src && touch src/Generated.hpp && make Generated

Output:

[100%] Built target Generated

You see that make has nothing to do for the Generated target, because it now sees an OUTPUT of your custom command that is newer than its dependencies. Of course, that touched file isn't the generated one; we need to bring it all together.

Solution

Don't write to your source tree.

Since ${PROJECT_BINARY_DIR}/src does not exist, you need to either create it, or live with the created files on the top dir. I did the latter for simplicity here. I also removed unnecessary ${PROJECT_SOURCE_DIR} uses.

add_custom_command(
    OUTPUT Generated.hpp
    COMMAND ${PROJECT_SOURCE_DIR}/generator.sh \
            ${PROJECT_SOURCE_DIR}/Resources/data.txt \
            > Generated.hpp
    DEPENDS Resources/data.txt
    DEPENDS generator.sh
)

Don't glob, keep control over what actually gets compiled:

add_executable( ${PROJECT_NAME} src/main.cpp )

Add the binary tree to your target's include path. After add_executable:

target_include_directories( ${PROJECT_NAME} PRIVATE ${PROJECT_BINARY_DIR} )

That's it, things work as expected now.