Hypothesis: test defensive programming

Question

I'm writing tests with Hypothesis to test functions with restricted range. Consider the following toy example:

import math
from hypothesis import assume, example, given

def inc(value: float) -> float:
    assert not math.isnan(value)
    return value + 1

@given(st.floats())
def test_inc(value: float):
    assume(not math.isnan(value))
    if(value == math.inf or value == -math.inf):
        assert inc(value) == value
    else:
        assert inc(value) > value

Here I'm testing the behaviour for finite and infinite inputs, but not NaN. I would like to test that out-of-range values, such as NaN, correctly throw an AssertionError.

Is there a pattern built into Hypothesis where I can test this? Ideally I'd like an approach where I don't compromise the coverage of within-range values.

Answer 1

If you want to manually check that you're generating some values in each of a range of buckets, you can use the hypothesis.event() function and then investigate via CLI statistics reporting or dig as deep as you like with observability dumps - or even fail CI with a post-test script if the *_testcases.jsonl file shows you're missing something.

If you want your tests to fail unless at least one value can be found in each bucket, you'll need a separate test (or at least parametrize case) for each such bucket. Defining helper functions (e.g.) might be useful; you'll probably also want derandomize=True to avoid flakiness for rare buckets.