Rust Reqwest 10x Slower Than Postman & Firefox

Question

I was testing the performance of reqwest in Rust but somehow the response.text().await is taking a good 1-3 seconds to complete. I have tested the exact same page with postman and firefox and it only takes 100-300ms. I have tried multiple times and every single time Postman & Firefox are faster.

I have tried using Hyper as well with .aggregate() and have tried using reqwest's .bytes() and .chunk() and they are all just as slow. The interesting thing is that .send().await takes the normal 100-300ms as it's supposed to. Calling .content_length() is also perfectly efficient. I also tried using Javascript's fetch() and calling await response.json(), but this is most certainly faster.

Also this has nothing to do with the release build as manually running the release exe is just as slow. Rust and reqwest are also fully up to date.

Here is the code:

use std::time::Instant;

#[tokio::main]
async fn main() {
    let client = reqwest::Client::new();

    let response = client
        .get("https://registry.npmjs.org/react")
        .send()
        .await
        .unwrap();

    let now = Instant::now();
    let data = response.text().await;
    let elapsed = now.elapsed();

    println!("{}ms", elapsed.as_millis());
}

Using reqwest::blocking also does not make a difference:

use std::time::Instant;

fn main() {
    let now = Instant::now();
    let data = reqwest::blocking::get("https://registry.npmjs.org/react")
        .unwrap()
        .text()
        .unwrap();
    let elapsed = now.elapsed();

    println!("{}ms", elapsed.as_millis());
}

The time the request takes does decrease if I request an endpoint with less data. For example https://registry.npmjs.org/react/is-even takes 300ms with reqwest, but still 70ms with Postman.

Any ideas what's causing this?

Answer 1

There are actually a lot of factors that lead to this unexpected result. Here are our findings.

• You can use the application/vnd.npm.install-v1+json; q=1.0, application/json; q=0.8, */* header (as per the NPM Registry Docs) to significantly reduce the size of the payload and make the request faster. You can also do Content-Encoding: gzip however this is quite a bit slower.
• There is likely some server side caching involved with the Firefox headers which is possibly why it is so fast even when browser caching is turned off. When clearing the cache and cookies an expected response time is achieved.
• It is possible that because I live in a remote area, the closest distribution center is quite far away making my response times a bit slower
• Using GET·/{package}/{version} instead of GET·/{package} in addition to the header is a far better solution if possible in your application.

Note that these findings are not 100% certain but are speculation. I will update this answer if I find more information. Happy coding!