Remove outdated intermediate files before the build

I have a project where a lot of the source files needs to be modified by a script before they are compiled. The build process has 2 steps:

Run a script on the original sources to create intermediate sources.
Compile the intermediate sources.

It works fine when the original source files is modified or a new one is created. In such cases SCons is able to build / rebuild the appropriate files.

However, when a source file is deleted, the corresponding intermediate file is not removed, which may end in successful build where it should fail due to missing source.

Example:

SConscript:

env = Environment()

source_files = ['main.cc.template', 'some-header.hh.template']

def make_intermediate(env, source):
    target = source[:-9] # Remove ".template" from the file name
    return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])

env.Program('my-program', intermediates)

main.cc.template:

#include "some-header.hh"

int main() {
    return get_number();
}

some-header.hh.template:

inline int get_number() {
    return 0;
}

This compiles correctly but if you remove the file some-header.hh.template from the list and from the filesystem, it still compiles while it shouldn't.

You need to manually remove the intermediate file some-header.hh from the file system or else you'll get a false-positive result of build and subsequent tests. I would like to automate the deletion process to prevent inevitable broken commits that will happen if I won't.

I've managed to create a dirty solution of the problem:

env = Environment()

source_files = ['main.cc.template']

def make_intermediate(env, source):
    target = source[:-9] # Remove ".template" from the file name
    return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])

# --- The new starts code here ---
old_intermediates = Glob('*.hh') + Glob('*.cc')
intermediates_to_delete = [x for x in old_intermediates if x not in intermediates]
for x in intermediates_to_delete:
    x.remove()
# --- The new code ends here ---

env.Program('my-program', intermediates)

This more or less works. However, the files are removed too late and SCons seem to already be aware of their presence which causes the build error to origin from SCons and not the C++ compiler. Because of that, the error message is less helpful. Also, I don't know if such operations are good for the stability of the SCons itself.

The error I'm getting is:

scons: *** [main.o] /home/piotr/tmp/some-header.hh: No such file or directory

Is there a clear way to delete outdated intermediate files?

Solution

Your approach is more or less correct. SCons doesn't have any built-in mechanism to remove such dangling intermediate files; you need to write your own.

The error you're getting is caused by the fact you've used the SCons function Glob. It creates File nodes and makes SCons aware of existence of those files. (Btw, the SCons function remove() is not designed to be called outside of a builder; you shouldn't do that.)

To avoid the problem, you need to delete the file before SCons has a chance to search for it. You can just replace SCons function with standard Python library, like pathlib. (It will require some tinkering to convert intermediates to pathlib objects too, but it won't be that much more code.)

A fixed solution:

env = Environment()

source_files = ['main.cc.template']

def make_intermediate(env, source):
    target = source[:-9] # Remove ".template" from the file name
    return env.Command(target, source, Copy("$TARGET", "$SOURCE")) # Modify the source
env.AddMethod(make_intermediate, 'MakeIntermediate')
intermediates = Flatten([env.MakeIntermediate(x) for x in source_files])

# --- The new starts code here ---
from pathlib import Path
old_intermediates = list(Path.cwd().glob('*.hh')) + list(Path.cwd().glob('*.cc'))
current_intermediates = [Path(x.get_path()).resolve() for x in intermediates]
intermediates_to_delete = [x for x in old_intermediates if x.resolve() not in current_intermediates]
for x in intermediates_to_delete:
    print('unlink:', x)
    x.unlink()
# --- The new code ends here ---

env.Program('my-program', intermediates)

This gives the expected error message:

main.cc:1:10: fatal error: some-header.hh: No such file or directory
    1 | #include "some-header.hh"
      |          ^~~~~~~~~~~~~~~~
compilation terminated.
scons: *** [main.o] Error 1