Catching a requests.exceptions.HTTPError in pytest

Question

I have a method that makes a call into another method which does an API call and then does raise_for_status. I want to test this method, which looks like this:

async def _get_deal(self, reference: DtoDealReference) -> Deal:
    try:
        deal = await self._trades_client.get_deal(reference.platform, reference.uniqueness_hash, reference.version)
    except requests.exceptions.HTTPError as exc:
        if exc.response.status_code == 404:
            raise MissingDealError(reference) from exc
        raise
    return deal

This issue is that, when testing this using pytest, I get the following error on the catch:

TypeError: catching classes that do not inherit from BaseException is not allowed

I find this odd because HTTPError inherits from RequestException, which itself inherits from Exception. So, unless the conditions which would raise this TypeError are written poorly, this shouldn't be an issue.

For reference, here is my test:

@pytest.mark.anyio
@patch("service_valuation.infrastructure.ports.trade_client.Session.get")
async def test_short_term_ledger_entry_converter_to_data_deal_reference_not_found(
    mock_get, trades_client: TradesClient, result_entities: List[Entity]):

    mock_entity_response = MagicMock()
    mock_entity_response.json.return_value = TypeAdapter(List[Entity]).dump_python(result_entities, by_alias=True)
    mock_entity_response.raise_for_status = MagicMock()
    mock_deal_response = Response()
    mock_deal_response.status_code = 404
    mock_get.side_effect = [mock_entity_response, mock_deal_response]

    converter = EntryConverter(deal_converter=DealReferenceConverter(), trades_client=trades_client)

    dto = LedgerEntry(
        id=1,
        valuation_type=ValuationType.REVENUE,
        status=Status.BILLED,
        description="test",
        amount=100,
        counterparty="EEX",
        deal=DealReference(uniqueness_hash="test", platform="EEX", version=1),
    )

    with pytest.raises(MissingDealError) as ex_info:
        await converter.to_data(dto)

    # Finally, verify that the error is as expected
    assert ex_info.type is MissingDealError
    assert ex_info.value.reference == DealReference(uniqueness_hash="test", platform="EEX", version=1)

Does anyone know why this is happening and what I can do to fix it?

Answer 1

It turns out the issue was in a decorator on the _get_deal method. To give additional context, I'll provide the signature of this method here:

@on_exception(expo, [Timeout, ConnectionError], max_tries=6)
async def get_deal(self, platform: str, uniqueness_hash: str, version: int) -> Deal:

Specifically, the issue was in the on_exception decorator which I imported from the backoff package. This decorator calls _async.retry_exception with my provided exception types, as an argument named exception. Inside this method you will see the following line:

except excetion as e:

And there is the problem. When I originally decorated on_deal, I provided a list of exception types rather than a single exception type or a tuple. As such, when I was testing around the call to raise_for_status inside on_deal, an exception was being raised and calling the catch block inside _async.retry_exception to check the type of exception, which was a list. Since this is invalid, it raised the exception I actually saw.

Fortunately, the fix was to simply change the decorator to be a tuple instead of a list (notice the parentheses instead of the square brackets):

@on_exception(expo, (Timeout, ConnectionError), max_tries=6)