Hypothesis does not seem to honor the maxfail argument of pytest

Question

In spite of specifying maxfail=1, hypothesis seems to continue generating examples and running them and failing much later.

Is there a workaround?

Here is a small example:

from hypothesis.stateful import invariant, rule, RuleBasedStateMachine


class MaxFail(RuleBasedStateMachine):
    count = 0

    @rule()
    def process(self):
        self.count += 1

    @invariant()
    def all_done(self):
        print('-- in invariant %d' % self.count)
        if self.count > 1:
            assert False


MaxFailTest = MaxFail.TestCase

Answer 1

This is because from Pytest's point of view the whole stateful test is only one test - it invokes MaxFailTest.runTest(), and if that fails it will not run any other test functions.

On the other hand, Hypothesis doesn't know about any Pytest arguments or settings beyond those added in its plugin. It can be used equally well with pytest, or unittest, or any other test runner because it simply wraps the inner test function you wrote.

In short: Hypothesis doesn't know about the --maxfail argument, and Pytest doesn't know the test will fail until Hypothesis raises an error with the minimal example(s) it found.