Hypothesis stateful testing with pytest.raises doesn't report sequence of steps

I want to write a hypothesis.stateful.RuleBasedStateMachine which asserts that an exception is raised under certain circumstances. pytest provides the raises context manager for writing tests about exceptions. If I use pytest.raises inside a hypothesis.stateful.rule, the sequence of steps that lead to the test failure is not reported.

Rewriting the rule without pytest.raises results in the desired behaviour: the sequence of steps is displayed.

Here is some sample code:

from os import getenv

from pytest import raises

from hypothesis.stateful   import RuleBasedStateMachine, rule

SHOW_PROBLEM = getenv('SHOW_PROBLEM') == 'yes'


# A state machine which asserts that an exception is raised in under some condition
class FifthCallShouldRaiseValueError(RuleBasedStateMachine):

    def __init__(self):
        super().__init__()
        self.model = Model()
        self.count = 0

    if SHOW_PROBLEM:

        # This version does NOT report the rule sequence
        @rule()
        def the_rule(self):
            self.count += 1
            if self.count > 4:
                with raises(ValueError):
                    self.model.method()

    else:

        # This version DOES report the rule sequence
        @rule()
        def the_rule(self):
            self.count += 1
            if self.count > 4:
                try:
                    self.model.method()
                except ValueError: assert True
                except           : assert False
                else             : assert False


T = FifthCallShouldRaiseValueError.TestCase


# A model that deliberately fails the test, triggering reporting of
# the sequence of steps which lead to the failure.
class Model:

    def __init__(self):
        self._count = 0

    def method(self):
        self._count += 1
        if self._count > 4:
            # Deliberate mistake: raise wrong exception type
            raise TypeError

To observe the difference in behaviour, exectue the test with

SHOW_PROBLEM=yes pytest <...>
SHOW_PROBLEM=no pytest <...>

In the second case the output will show

state = FifthCallShouldRaiseValueError()
state.the_rule()
state.the_rule()
state.the_rule()
state.the_rule()
state.the_rule()
state.teardown()

This sequence of steps is missing from the output in the first case. This is usdesirable: the sequence should be show in both cases.

pytest.raises raises Failed: DID NOT RAISE <class 'ValueError'> while the hand-written version raises AssertionError. The former is more informative when it comes to the failure to raise the required exception, but somehow seems to prevent hypothesis.stateful from reporting the sequence of steps, which tells us how we got into that state, and is often the most interesting part of the output.

What can be done to mitigate this, i.e. to ensure that the sequence of steps is printed out, besides not using pytest.raises?

Solution

It turns out that the steps are not printed if a rule raises BaseException or a non-Exception subclass. pytest.raises(...) raises just such an error if it doesn't get the expected exception, and there you are.

https://github.com/HypothesisWorks/hypothesis/issues/1372

It's not a particularly gnarly bug now that it's been identified - and thanks for your part in that, by reporting a reproducible case! - so we should get a fix out soon.

Update: this bug was fixed in Hypothesis 3.65.1, on 2018-07-03.