Realtime output from subprocess in Python

Question

I am running a Django webserver, and am trying to print the output of a subprocess in real-time to the log. If I commend out the readline line and uncomment the yowzer line, I get the desired stream of logged output, but as soon as I try to run this as-is I get no output. What am I doing wrong? All the results I can find seem to suggest that this would work as-is.

sp = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE)

while sp.poll() is None:
    logging.critical(sp.stdout.readline())
    #logging.critical('yowzer')
    time.sleep(10)

Thank you!

[edit April 3] It turns out the issue wasn't that the command was long-running, but rather that I was using a semicolon to concat a 'su user' command with the main command. Removing the su solved the issue.

Answer 1

Getting (or rather, not getting) real time output is a perennial annoyance when calling subprocesses in Python. See this question for an in-depth discussion, but the short version is that adding the keyword arg bufsize=1 to the Popen call may be the secret sauce you need.

Also, why do you have time.sleep(10)? You do realize that means that you'll have to wait 10 seconds in between each line getting logged, right? Try restructuring the calls you're making to read the output. Assuming you're using Python 3, try this:

from subprocess import PIPE, Popen

with Popen(command, shell=True, stdout=PIPE, bufsize=1) as sp:
    for line in sp.stdout:
        logging.critical(line)

I just tested the above code and can confirm it works. Try it out yourself here. The linked test version prints a timestamp before each line of the log.

If that's still not working

There's nothing you can do in the Python script if the subprocess you're calling doesn't flush it's output on a regular basis. You mentioned Django, right? Is the output you're trying to log being produced via standard Python print() calls? If so, you can modify them to flush more aggressively:

print(s, flush=True)

Every other language I've worked with also has an equivalent flush command, so (assuming you have access to/can modify the source of the subprocess) you should be able to make this work.