How to allow non mocked endpoints when using responses library

In Python it's really common to use requests library to make your HTTP requests.

import requests

r = requests.get('http://google.com')
print(f"Code: {r.status_code}\nText head: {r.text[:100]}")

>>> Code: 200
>>> Text head: <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="es"><head><meta content

Then, it's also really common to mock HTTP requests using responses library.

import requests
import responses

@responses.activate
def my_request():
    responses.add(responses.GET, 'http://google.com', body="Hi! Mocked :)")
    r = requests.get('http://google.com')
    return r

r = my_request()
print(f"Code: {r.status_code}\nText head: {r.text[:100]}")

>>> Code: 200
>>> Text head: Hi! Mocked :)

However, I've an use case in which I need an endpoint to be mocked, and another one to be actually used.

E.g. Actually access the 'http://google.com' endpoint to run the function, then mock other data from another endpoint and test my function.

However, in this situation I found that the endpoint that I would actually want to visit fails just because is not available within responses current mocked endpoints.

import requests
import responses

@responses.activate
def my_request():
    responses.add(responses.GET, 'http://google.com', body="Hi! Mocked :)")
    # To be mocked
    r1 = requests.get('http://google.com')
    # To actually access
    r2 = requests.get('http://bing.com')
    return r1, r2

r1, r2 = my_request()
print(f"Code r1: {r1.status_code}\nCode r2: {r2.status_code}")

Produces as output:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 3, in wrapper
  File "<stdin>", line 7, in my_request
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/responses/__init__.py", line 765, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/responses/__init__.py", line 745, in _on_request
    raise response
requests.exceptions.ConnectionError: Connection refused by Responses - the call doesn't match any registered mock.

Request:
- GET http://bing.com/

Available matches:
- GET http://google.com/ URL does not match

How can I make responses to allow reaching some actual non-mocked endpoints?

Solution

responses library has a mechanism to allow this using the add_passthru function. Its behaviour is similar to the add function but it just needs the endpoint and no further data.

It's also important to notice that if the endpoint resolves to a different URL, it won't work. For example, adding it to the example:

import requests
import responses

@responses.activate
def my_request():
    responses.add(responses.GET, 'http://google.com', body="Hi! Mocked :)")
    responses.add_passthru('http://bing.com')
    # To be mocked
    r1 = requests.get('http://google.com')
    # To actually access
    r2 = requests.get('http://bing.com')
    return r1, r2

r1, r2 = my_request()
print(f"Code r1: {r1.status_code}\nCode r2: {r2.status_code}")

Will produce an error because r2 request will try to connect to http://www.bing.com, and the endpoint was defined without the www: http://bing.com

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 3, in wrapper
  File "<stdin>", line 8, in my_request
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 677, in send
    history = [resp for resp in gen]
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 677, in <listcomp>
    history = [resp for resp in gen]
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects
    resp = self.send(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/responses/__init__.py", line 765, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/responses/__init__.py", line 745, in _on_request
    raise response
requests.exceptions.ConnectionError: Connection refused by Responses - the call doesn't match any registered mock.

Request:
- GET http://www.bing.com/

Modifying the passthru defined endpoint will solve this issue:

    # responses.add_passthru('http://bing.com')
    responses.add_passthru('http://www.bing.com')