Handling of pydantic ValidationError when testing with hypothesis.given

Question

When using hypothesis to test my pydantic models, I do not know how to handle ValidationError thrown by custom validators. This is a very small example that shows the problem:

# model
from pydantic import BaseModel, validator

class SimpleModel(BaseModel):
    a: int
    b: int

    @validator('b')
    def check_numbers(cls, b, values):
        if b*values['a'] < 0:
            raise ValueError('a*b > 0 does not hold')
        return b

# test
from hypothesis import given, strategies as st

@given(st.builds(SimpleModel))
def test_simple_model(instance: SimpleModel):
    assert type(instance.b) == int

Up to now I have written custom hypothesis search strategies to only generate instances that are valid. But this gets very tedious for more complex models, so in my opinion there has to be a smarter way to "use" the ValidationError. The error is also raised before the test function, therefore I cannot handle it in the test function.

I would need a possibility to generate instances, that simply skips instances that raise a ValidationError.

Answer 1

After using more of the functionality of hypothesis, I came up with this approach, which uses composite strategy and assume. With composite I create a custom strategy to be used in the given decorator. Inside assume is used to tell hypothesis that examples are bad and should be skipped whenever a ValidationError is thrown.

from pydantic import ValidationError
from hypothesis import given, assume, strategies as st

@st.composite
def simple_model_strategy(draw) -> SimpleModel:
    try:
        simple_model = draw(st.builds(SimpleModel))
    except ValidationError:
        assume(False)
    return simple_model

@given(simple_model_strategy())
def test_simple_model(instance: SimpleModel):
    assert type(instance.b) == int

Notes:

• I used assume in the test before, but like this it can also be used during model generation.
• If assume is called on most of the autogenerated instances, hypothesis warns about problems with the test case.