Search code examples
pythonbuildmultiprocessingscons

Combination of chdir=1 and num_jobs>1 in SCons

I have a rather lengthy test task which is automated by SCons and could be parallelized. However, it currently relies on using chdir=1, which at the moment is not trivial to remove. Now, as soon as I use -j2 respectively SetOption('num_jobs', 2) the job fails and the following minimal (non-) working sample shows why: The chdir is not applied on a per-thread basis but affects all jobs at once.

Is this behaviour inteded? And is there any way to prevent this?

# File SConstruct
import os, time

def my_build_fun(target, source, env):
    for i in range(1, 5):
        out = "my_build_fun: %d %s %s" % (i, str(source[0]), os.getcwd())
        print out
        time.sleep(0.5)
    return None

bld = Builder(action = my_build_fun,
              suffix = '.output',
              src_suffix = '.input',
              chdir=1)
env = Environment(BUILDERS = {'Foo' : bld})

Alias('do_a', env.Foo('folder_a/do_a'))
Alias('do_b', env.Foo('folder_b/do_b'))

Default(['do_a', 'do_b'])

Operating on 

+--- SConstruct
+-+- folder_a
| \--- do_a.input
\-+- folder_b
  \--- do_b.input

Result for scons -j1: As expected

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
os.chdir('folder_a')
my_build_fun(["folder_a\do_a.output"], ["folder_a\do_a.input"])
my_build_fun: 1 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 2 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 3 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 4 folder_a\do_a.input H:\Playground\folder_a
os.chdir('H:\\Playground')
os.chdir('folder_b')
my_build_fun(["folder_b\do_b.output"], ["folder_b\do_b.input"])
my_build_fun: 1 folder_b\do_b.input H:\Playground\folder_b
my_build_fun: 2 folder_b\do_b.input H:\Playground\folder_b
my_build_fun: 3 folder_b\do_b.input H:\Playground\folder_b
my_build_fun: 4 folder_b\do_b.input H:\Playground\folder_b
os.chdir('H:\\Playground')
scons: done building targets.

Result for scons -j2: Trying to chdir into .\folder_a\folder_b which indeed does not exist.

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
os.chdir('folder_a')
my_build_fun(["folder_a\do_a.output"], ["folder_a\do_a.input"])
my_build_fun: 1 folder_a\do_a.input H:\Playground\folder_a
os.chdir('folder_b')
my_build_fun(["folder_b\do_b.output"], ["folder_b\do_b.input"])
scons: *** [folder_b\do_b.output] folder_b: The system cannot find the file specified
my_build_fun: 2 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 3 folder_a\do_a.input H:\Playground\folder_a
my_build_fun: 4 folder_a\do_a.input H:\Playground\folder_a
my_build_fun
os.chdir('H:\\Playground')
scons: building terminated because of errors.
Solution

  • From the SCons man page

    http://www.scons.org/doc/2.3.2/HTML/scons-man.html

    WARNING: Python only keeps one current directory location for all of the threads. This means that use of the chdir argument will not work with the SCons -j option, because individual worker threads spawned by SCons interfere with each other when they start changing directory.

    So if your script cannot be run from the base directory of the build. The (current) simplest way to solve that is to just create a script which will cd and then run your command. Or even put it in the shell command executed 

    env.Command(b,a,"cd ${SOURCE.dir}; do whatever -o ${TARGET.file} -i ${SOURCE.file}").