stop pytest tests if several tests have a specific exception

Question

I want to stop the tests suite using pytest.exit() if any of the tests fails with a specific exception.

For example: 50 tests, any of them can fail at some point with that exception, I want to stop the execution if at least 2 of these tests fail over this exception.

I've tried to keep a global counter ( a fixture with scope='session') between tests and update it every time I catch this exception, but was unable to keep it's value between tests.

any ideas?

Answer 1

This is possible through the use of Hooks.

Specifically we will take advantage of two specific hooks, pytest_sessionstart and pytest_exception_interact. We will use pytest_sessionstart to keep track of the number of specific exceptions we are willing to tolerate, think of this as the place to store the "global counter" that you mention. The other hook, pytest_exception_interact will be used to interact with failed tests to check the type of exception that was returned, if any.

Create a conftest.py file in the root directory of your test folder and place the following:

import pytest


EXC_MAP = {
        ZeroDivisionError: 1,
        KeyError: 1
    }


def pytest_sessionstart(session):
    session.__exc_limits = EXC_MAP


def pytest_exception_interact(node, call, report):
    session = node.session
    type_ = call.excinfo.type

    if type_ in session.__exc_limits:
        if session.__exc_limits[type_] == 0:
            pytest.exit(f"Reached max exception for type: {type_}")
        else:
            session.__exc_limits[type_] -= 1

The EXC_MAP is the map of exceptions --> failures that we are willing to allow in our test invocation. In the pytest_sessionstart hook we set a private variable on the session to keep track of these variables. In the pytest_exception_interact hook we get the type of exception that was raised by the test, we check it against the thresholds, and if the count for that exception has reached 0, we exit from pytest, skipping the remained of the tests.

Below is an example test script and the output in the terminal.

def foo(a, b):
    return a / b


def test_foo():
    result = foo(1, 0)
    assert result == 1


def test_foo1():
    result = foo(1, 1)
    assert result == 1


def test_foo2():
    result = foo(1, 0)
    assert result == 1


def test_foo3():
    result = foo(1, 1)
    assert result == 1

And when running these the terminal shows:

collected 4 items                                                                                                                                            

test_foo.py F.F

========================================================================== FAILURES ==========================================================================
__________________________________________________________________________ test_foo __________________________________________________________________________

    def test_foo():
>       result = foo(1, 0)

test_foo.py:6: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

a = 1, b = 0

    def foo(a, b):
>       return a / b
E       ZeroDivisionError: division by zero

test_foo.py:2: ZeroDivisionError
_________________________________________________________________________ test_foo2 __________________________________________________________________________

    def test_foo2():
>       result = foo(1, 0)

test_foo.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

a = 1, b = 0

    def foo(a, b):
>       return a / b
E       ZeroDivisionError: division by zero

test_foo.py:2: ZeroDivisionError
================================================================== short test summary info ===================================================================
FAILED test_foo.py::test_foo - ZeroDivisionError: division by zero
FAILED test_foo.py::test_foo2 - ZeroDivisionError: division by zero
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Reached max exception for type: <class 'ZeroDivisionError'> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================ 2 failed, 1 passed in 0.20s =================================================================

We can see that it collected all 4 tests, but only ran 3 of them because the threshold for ZeroDivisionError was reached prior to running the final test.