Search code examples
pythondockersubprocessbackground-process

Terminating docker container on SIGTERM on warping script


I have the following Python script that warps a docker container run:

import subprocess
import sys
import signal

container_id = None

def catch_term(signum, frame):
    sys.stderr.write('Caught SIGTERM. Stopping container\n')
    if container_id != None:
        cmd = ['docker', 'rm', '-f', container_id]
        subprocess.call(cmd)
    sys.exit(143)

signal.signal(signal.SIGTERM, catch_term)
cmd = ['docker', 'run', '--cidfile', cidfile, '-i', image_name, container_process_cmd]
p = subprocess.Popen(cmd)
# This function waits for the file to contain 64 bytes and then reads the container id from it
container_id = get_container_id_from_cidfile(cidfile)
p.communicate()
sys.exit(p.returncode)

When I run the Python process on interactive mode, and kill that warping script during the run (kill <PID>), the catch_term function is called and the container is removed. Then my process exits.

However, when I run the warping process on the background, i.e. my_warping_script_above.py &, and kill it in the same way, the catch_term function is not called at all and my script keeps running, including the container.

  • Any idea how to make the catch_term function be called even when the script runs on background?

  • I think it has something to do with the fact that docker run is a "protected" process. It should only be able to get killed using a docker command, such as docker rm -f <CONTAINER ID>, or using Ctrl+C if called with -i. That's the reason why I'm trying to catch the TERM signal and call docker rm, but my function is not called when the script runs on background... (I think it tries to first kill the child process, then as it fails, it does not reach to stage of calling my registered function).


Solution

  • Answering myself:

    This trick solved my problem: adding preexec_fn=os.setsid to the subprocess.Popen function:

    p = subprocess.Popen(cmd, preexec_fn=os.setsid)