How to separate warnings from errors found in stderr when using Popen.communicate?

I used Python's subprocess.Popen to execute a command and capture its output:

p = Popen(cmd, stdout=PIPE, stderr=PIPE,shell=True)
stdout, stderr = p.communicate()

I want to use stderr to tell users when there was an error and exit my script:

if stderr !='':
    return {'error':stderr}

But now I've found that stderr can contain warnings that could be safely ignored, so my script should not exit, but continue to finish the job.

Is there a way to separate warnings from errors in stderr?

Solution

univerio is correct, in that there is no specific meaning for any bytes you discover in stderr... think of it as "standard not-output" instead of "standard error". However, on most operating systems, you can use the process's exit status (or "return code") to skip most of the progress bars and other non-error output.

A Popen object has a field called returncode, which is used to store whatever value the subprocess returned when it exited. This value is None until 1) the process terminates, and 2) you collect its exit status with either the poll or wait methods (at least on Unix-like systems). Since you're using communicate, which always does both 1 and 2, p.returncode should always be an integer by the time you care about it.

As a general rule, a 0 exit status indicates success, while any other value indicates failure. If you trust the programs you're calling to return proper values, you can use this to skip most of the junk output on stderr:

# ...same as before...
stdout, stderr = p.communicate()
if p.returncode and stderr:
    return {'error': stderr}

If the bytes found in stderr weren't important enough to produce a non-0 exit status, they're not important enough for you to report, either.

To test this, you can write a few tiny scripts that produce stderr output and then exit, either successfully or not.

warnings.py:

import sys
print('This is spam on stderr.', file=sys.stderr)
sys.exit(0)

errors.py:

import sys
print('This is a real error message.', file=sys.stderr)
sys.exit(1)

This still leaves the task of separating spinning batons and other progress-report spam from actual error messages, but you'll only have to do that for processes that failed... and maybe not even then, since the "not dead yet!" messages might actually be useful in those cases.

PS: In Python 3, stdout and stderr will be bytes objects, so you'll want to decode them before treating them like strings.