Search code examples
pythonpyteststderr

Can pytest throw stderr on failure?


I'm running pytest through a Python script, and I want it to know if the pytest has failed. The intuitive way for me would be to catch any stderr on pytest's exit, but I just can't seem to find a way to do it. Any tips?

My tests look like this:

def test_2():
  object = Object()
  object.func()
  assert object.property == expected_value

The tests are called like this:

pytest.main([path_to_test, "-s"])

I want to know whether even a single assert has failed, and do something about it, maybe like so:

err = io.StringIO()
with contextlib.redirect_stderr(err):
  pytest.main([path_to_test, "-s"])

So basically I want pytest to throw anything in stderr when it's done if any test has failed.

A solution I've found is to replace asserts with a function that immediately throws an error, but then not all tests are ran.

Edit: OK, I just had another idea, which looks like this:

def assert_test(test_case):
  if not test_case:
      sys.stderr.write("failure")
  assert test_case

I guess I found something that works for me, but I still find it weird that there's no built-in or prettier solution...

Edit edit: OK, so that solution messes up the logging. I've resorted to monkey patching the assert... IDK if that's a great idea.

class AssertionError():
def __init__(self, *args, **kwargs):
    sys.stderr.write("failure")
    raise builtins.AssertionError

Now the calling script is aware if there was a failure.


Solution

  • pytest.main() returns the exit code as a result, so if you're looking for a pass/fail on the entire run you can check the return value, just like you would in a shell script. It is very coarsely-grained, however, it is just the one pass/fail value for the entire test run.

    According to the docs, the following exit codes are defined:

    • Exit code 0: All tests were collected and passed successfully
    • Exit code 1: Tests were collected and run but some of the tests failed
    • Exit code 2: Test execution was interrupted by the user
    • Exit code 3: Internal error happened while executing tests
    • Exit code 4: pytest command line usage error
    • Exit code 5: No tests were collected