Mocked URLs not being forward to handler when using requests_mock in another fixture

Question

I am creating my own fixture to simulate a service endpoint needed in my unit tests. In order to intercept the HTTP requests, I use requests_mock as follows:

@pytest.fixture
def sparql_endpoint(requests_mock):
    yield lambda uri, initial_data: Endpoint(requests_mock, uri, initial_data)

and in Endpoint.__init__ I do the following:

    m.post(url=uri, raw=self.handle_post)
    m.get(url=uri, raw=self.handle_get)

In my actual testcase I inject the endpoint and initialize it:

def test_basic_select(my_endpoint):
    repo_uri = 'https://my.rdfdb.com/repo/sparql'
    rdf_files = ['tests/upper_ontology.ttl',
                 'tests/domain_ontology.ttl',
                 'tests/instance_data.ttl']
    endpoint = sparql_endpoint(repo_uri, rdf_files)

Which does, in fact, initialize the mocked endpoint and I see Mocker.start() get invoked if I set a breakpoint there. However, later in the testcase I get the following:

..\..\AppData\Local\Programs\Python\Python37\lib\urllib\request.py:222: in urlopen
    return opener.open(url, data, timeout)
..\..\AppData\Local\Programs\Python\Python37\lib\urllib\request.py:525: in open
    response = self._open(req, data)
..\..\AppData\Local\Programs\Python\Python37\lib\urllib\request.py:543: in _open
    '_open', req)
..\..\AppData\Local\Programs\Python\Python37\lib\urllib\request.py:503: in _call_chain
    result = func(*args)
..\..\AppData\Local\Programs\Python\Python37\lib\urllib\request.py:1360: in https_open
    context=self._context, check_hostname=self._check_hostname)

E               urllib.error.URLError: <urlopen error [Errno 11001] getaddrinfo failed>

..\..\AppData\Local\Programs\Python\Python37\lib\urllib\request.py:1319: URLError

Because it cannot resolve the fake URL I gave it. So, did I somehow mess up the handler registration so that the Matcher is not kicking the request there? Why is urlopen still trying to resolve the host?

Answer 1

I have figured out the issue, which was due to the underlying code (SPARQLWrapper) not using requests, instead using urlopen directly, and thus bypassing my mock. In order to be able to intercept both types of access, I resorted to using HTTPretty, which did a more thorough mocking. The code ended up looking as follows:

First, in the fixture itself, I enabled the mocking:

    httpretty.set_default_thread_timeout(60)
    # enable HTTPretty so that it will monkey patch the socket module
    httpretty.enable(verbose=True, allow_net_connect=False)

    yield lambda uri, initial_data, **kwargs: Endpoint(uri, initial_data, **kwargs)

    # disable afterwards, so that you will have no problems in code that uses that socket module
    httpretty.disable()
    # reset HTTPretty state (clean up registered urls and request history)
    httpretty.reset()

Note that I yield a lambda that creates the instance implementing the fixture, so after it is cleaned up, httpretty can clean up as well.

In the fixture initialization, I create the bindings:

        httpretty.register_uri(httpretty.GET, uri,
                               body=self.handle_get)
        httpretty.register_uri(httpretty.POST, uri,
                               body=self.handle_post)

When I inject the fixture, I had to make an extra call to actually create it:

def test_request_get(sparql_endpoint):
    repo_uri = 'https://my.rdfdb.com/repo/sparql'
    rdf_files = ['tests/upper_ontology.ttl',
                 'tests/domain_ontology.ttl',
                 'tests/instance_data.ttl']
    # This calls the lambda defined in the fixture
    endpoint = sparql_endpoint(repo_uri, rdf_files)