Search code examples
pythonasynchronouspython-asyncioaiohttp

Time cost difference between opening a new thread to run requests and aiohttp.ClientSession for async IO?


I understood aiohttp supports the async IO so it's completely single thread. But run_in_executor sort of starts a new thread. But I tested for a task with 1000 downloads, it seems the difference is rather insignificant. But I assume aiohttp should be much faster cause the thread cost. Did I do something wrong?

async def get(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            print(url, resp.status)
            print(url, await resp.text())

loop = asyncio.get_event_loop()     
tasks = [                           
    get("http://www.google.com"),
    get("http://www.google.com")
]
loop.run_until_complete(asyncio.wait(tasks))    
loop.close() 




async def get_via_thread(url):
    loop = asyncio.get_event_loop()
    try:
        response = await loop.run_in_executor(None, functools.partial(requests.get, url=url))

Solution

  • But I tested for a task with 1000 downloads, it seems the difference is rather insignificant.

    Problem probably somewhere in your benchmark. It's hard to say where exactly since you didn't provide one to reproduce :)

    As example, you can take a look at one recent question where OP tried to compare threads and coroutines and got no difference and answer where this result explained and fix provided.