filter outpout of process with potentially unlimted output, detect exit code and timeout after X

Question

I have some existing Django code running under uwsgi (under Linux) with threading disabled, that executes for some requests a subprocess, that I don't have any control over.

The normal operation is following:

• the subprocess runs for a rather short time and returns either an exit code of 0 or something different. The code will write some messages to stdout / stderr. The return code (exit code) will tell me whether the work was done correctly. if execution failed it would be good to gather stdout/stderr and log it to understand why things failed.

On rare occasions however the subprocess can encounter a so far not understood race condition and will do following.

• it will write repeatedly a specific message to stdout and stderr and loop and hang forever.

As I don't know whether there are any other race conditions, that could freeze the process with or without any output. Id' also like to add a timeout. (Though a solution, that adresses getting the return code and detecting the repeated message would be already a good achievement.

What I tried so far is:

import os
import select
import subprocess
import time

CMD = ["bash", "-c", "echo hello"]
def run_proc(cmd=CMD, timeout=10):
    """ run a subprocess, fetch (and analyze stdout / stderr) and
        detect if script runs too long
        and exit when script finished
    """

    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout = proc.stdout.fileno()
    stderr = proc.stderr.fileno()

    t0 = time.time()
    while True:
        if time.time() - t0 > timeout:
            print("TIMEOUT")
            break
        rc = proc.returncode
        print("RC", rc)
        if proc.returncode is not None:
            break
        to_rd, to_wr, to_x = select.select([stdout, stderr], [], [], 2)
        print(to_rd, to_wr, to_x)
        if to_rd:
            if stdout in to_rd:
                rdata = os.read(stdout, 100)
                print("S:", repr(rdata))
            if stderr in to_rd:
                edata = os.read(stderr, 100)
                print("E:", repr(edata))
    print(proc.returncode)

In fact I don't need stdout and stderr to be handled separately, but this didn't change anything

However when the subprocess finished its output something really strange happens.

the output of select tells me, that stdout and stderr can be read from, but when I read I get an empty string.
proc.returncode is still None

How could I fix my above code or how could I save my problem in a different way?

Answer 1

Check at least on Popen.poll():

def run_proc(cmd=CMD, timeout=10):
    """ run a subprocess, fetch (and analyze stdout / stderr) and
        detect if script runs too long
        and exit when script finished
    """

    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout = proc.stdout.fileno()
    stderr = proc.stderr.fileno()

    t0 = time.time()
    while True:
        returncode = proc.poll()
        print("RC", returnode)
        if returncode is not None:
            break

        if time.time() - t0 > timeout:
            print("TIMEOUT")
            # You need to kill the subprocess, break doesn't stop it!
            proc.terminate()
            # wait for the killed process to 'reap' the zombie
            proc.wait()
            break

        to_rd, to_wr, to_x = select.select([stdout, stderr], [], [], 2)
        print(to_rd, to_wr, to_x)
        if to_rd:
            if stdout in to_rd:
                rdata = os.read(stdout, 100)
                print("S:", repr(rdata))
            if stderr in to_rd:
                edata = os.read(stderr, 100)
                print("E:", repr(edata))
    print(returncode)

Out:

RC None
[3] [] []
S: b'hello\n'
RC None
[3, 5] [] []
S: b''
E: b''
0