Python function capturing subprocess stdout and stderr to log file

Question

There is a lot going on here in a small amount of code. I'll try to keep this concise.

I have a python function that runs an external program and tees both stdout and stderr to a log file.

I'm using doctest to test the function. I need to test the output capture functionality. The code below shows my attempt to write the function and the test. The test is failing with nothing written to the log file. I'm not sure if the problem is in the test or the code under test, or perhaps both. Suggestions?

from __future__ import print_function

import subprocess

def run(command_line, log_file):
    """
    # Verify stdout and stderr are both written to log file in chronological order
    >>> run("echo text to stdout; echo text to stderr 1>&2", "log")
    >>> f = open("log"); out = f.read(); f.close()
    >>> print(out.strip())
    text to stdout
    text to stderr
    """
    command_line = "set -o pipefail; " + command_line + " 2>&1 | tee " + log_file

    # Run command. Wait for command to complete. If the return code was zero then return, otherwise raise CalledProcessError
    subprocess.check_call(command_line, shell=True, executable="bash")

The test result:

$ python -m doctest testclass.py
text to stdout
text to stderr
**********************************************************************
File "testclass.py", line 10, in testclass.run
Failed example:
    print(out.strip())
Expected:
    text to stdout
    text to stderr
Got:
    <BLANKLINE>
**********************************************************************
1 items had failures:
   1 of   3 in testclass.run
***Test Failed*** 1 failures.

Answer 1

Since doing a subprocess.check_call with shell=True, with 2 stdout/stderr redirections and a tee is not the best way to execute a command and capture output (actually it's closest to the worst way), I'm not really surprised that it fails.

My solution would be to drop the set -o pipefail for starters (you don't need to check return code here) and wrap both commands in parentheses else redirection / tee only applies to the last one (I'm still puzzled why you get no output al all, to be honest, though):

command_line = "(" + command_line + ") 2>&1 | tee " + log_file

And if you had to restore the pipefail thing, do it within parentheses:

command_line = "(set -o pipefail; " + command_line + ") 2>&1 | tee " + log_file