Maintaining environment state between subprocess.Popen commands?

Question

I'm writing a deployment engine for our system, where each project specifies his custom deployment instructions.

The nodes are running on EC2.

One of the projects depends on a from source version of a 3rd party application.

Specifically:

cd /tmp
wget s3://.../tools/x264_20_12_2010.zip
unzip x264_20_12_2010.zip
cd x264_20_12_2010
./configure
make
checkinstall --pkgname=x264 --pkgversion "2:0.HEAD" --backup=no --deldoc=yes --fstrans=no --default

Currently I'm doing this with boto's ShellCommand (which uses subprocess.Popen internally), this looks something like this:

def deploy():
        ShellCommand("apt-get remove ffmpeg x264 libx264-dev")
        ShellCommand("apt-get update")
        ShellCommand("apt-get install -y build-essential checkinstall yasm texi2html libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-support libfaac-dev libjack-jackd2-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libsdl1.2-dev libtheora-dev libvorbis-dev libvpx-dev libx11-dev libxfixes-dev libxvidcore-dev zlib1g-dev")

        ShellCommand("cd /tmp")
        s3cmd_sync("s3://.../tools/x264_20_12_2010.zip", "/tmp/x264_20_12_2010.zip")
        ShellCommand("unzip x264_20_12_2010.zip")
        ShellCommand("cd x264_20_12_2010")
        ShellCommand("./configure")
        ShellCommand("make")
        ShellCommand(r'checkinstall --pkgname=x264 --pkgversion "2:0.HEAD" --backup=no --deldoc=yes --fstrans=no --default')

Sadly this fails, because cd /tmp applies to the subprocess, meaning that once I return the the father process and issue the second ShellCommand the exeenv is inherited from the father, which leans me to think that I need some execution framework for shell commands which will apply all commands in the same sub process without loosing context.

What is the recommend solution to this problem? Please note that loggings of the command line executed app is very important (how can you debug without it?), which is why I like ShellCommand... (see boto logging if interested).

Thank you,
Maxim.

Answer 1

I ended up doing this

def shell_script(appname, *commands):
        workspace = tempfile.mkdtemp(prefix=appname + '-')
        installer = open(workspace + "/installer.sh", 'w')

        installer.write("#!/bin/bash\n")
        installer.write("cd " + workspace + "\n")
        for line in commands:
                installer.write(line + "\n")
        ShellCommand("chmod u+x " + installer.name)
        installer.close()
        ShellCommand(installer.name)