Asyncio Making HTTP Requests Slower?

Question

I'm using Asyncio and Requests to benchmark a series of HTTP requests.

For some reason, it's slightly slower to use Asyncio than just straight Requests. Any idea why? Am I using Asyncio incorrectly?

import asyncio
import functools
import requests
import time

ts = time.time()
for i in range(10):
  @asyncio.coroutine
  def do_checks():
      loop = asyncio.get_event_loop()
      req = loop.run_in_executor(None, functools.partial(requests.get, "http://google.com", timeout=3))
      resp = yield from req
      print(resp.status_code)

  loop = asyncio.get_event_loop()
  loop.run_until_complete(do_checks())
te = time.time()
print("Version A: " + str(te - ts))

ts = time.time()
for i in range(10):
  r = requests.get("http://google.com", timeout=3)
  print(r.status_code)
te = time.time()

print("Version B:  " + str(te - ts))

Output:

Version A = Asyncio; Version B = Requests

200
200
200
200
200
200
200
200
200
200
Version A: 5.7215821743011475
200
200
200
200
200
200
200
200
200
200
Version B:  5.320340156555176

Answer 1

You are waiting for each request to finish before you start the next one. So you have the overhead of the event loop with no benefits.

Try this:

import asyncio
import functools
import requests
import time

ts = time.time()
loop = asyncio.get_event_loop()

@asyncio.coroutine
def do_checks():
    futures = []
    for i in range(10):
        futures.append(loop.run_in_executor(None, functools.partial(requests.get, "http://google.com", timeout=3)))

    for req in asyncio.as_completed(futures):
        resp = yield from req
        print(resp.status_code)

loop.run_until_complete(do_checks())
te = time.time()
print("Version A: " + str(te - ts))

ts = time.time()
for i in range(10):
    r = requests.get("http://google.com", timeout=3)
    print(r.status_code)
te = time.time()
print("Version B:  " + str(te - ts))

This is what I get when i run it:

$ python test.py 
200
...
Version A: 0.43438172340393066
200
...
Version B: 1.6541109085083008

Much faster, but really this is just spawning threads and waiting for the http library to finish, you don't need asyncio to do that.

You might want to checkout aiohttp as it was built for use with asyncio. requests is a fabulous library, but it is not made for asyncio.