Use requests-html to test Flask app without starting server

Question

I've been using the Flask test_client object to test my web applications. I used BeautifulSoup to parse the HTML output of some of these calls.

Now I wanted to try requests-html instead, but I cannot figure out how to make it work with the Flask test client. The examples all use the request package to get the response, but the Werkzeug test client does not make an actual HTTP call. From what I can tell, it sets up the environment and just calls the handler method.

Is there a way to make this work without having to have the actual service running?

Answer 1

requests-wsgi-adapter provides an adapter to mount a WSGI callable at a URL. You use session.mount() to mount adapters, so for requests-html you'd use HTMLSession instead and mount to that.

$ pip install flask requests-wsgi-adapter requests-html

from flask import Flask

app = Flask(__name__)

@app.route("/")
def index():
    return "<p>Hello, World!</p>"

from requests_html import HTMLSession
from wsgiadapter import WSGIAdapter

s = HTMLSession()
s.mount("http://test", WSGIAdapter(app))

r = s.get("http://test/")
assert r.html.find("p")[0].text == "Hello, World!"

The downside of using requests is that you have to add "http://test/" before every URL you want to make a request to. The Flask test client doesn't require this.

---

Instead of using requests and requests-html, you could also tell the Flask test client to return a Response that does the BeautifulSoup parsing for you. After a quick look at requests-html, I still prefer the direct Flask test client and BeautifulSoup API.

$ pip install flask beautifulsoup4 lxml

from flask.wrappers import Response
from werkzeug.utils import cached_property

class HTMLResponse(Response):
    @cached_property
    def html(self):
        return BeautifulSoup(self.get_data(), "lxml")

app.response_class = HTMLResponse
c = app.test_client()

r = c.get("/")
assert r.html.p.text == "Hello, World!"

---

You should also consider using HTTPX instead of requests. It's a modern, well maintained HTTP client library that shares many API similarities with requests. It also has great features like async, HTTP/2, and built-in ability to call WSGI applications directly.

$ pip install flask httpx

c = httpx.Client(app=app, base_url="http://test")

with c:
    r = c.get("/")
    html = BeautifulSoup(r.text)
    assert html.p.text == "Hello, World!"