Search code examples
pythonasynchronouspython-asyncioaiohttp

How to store responses from asyncio.gather() when semaphore is used?

This is the code I'm trying to run:

import asyncio
from aiohttp import ClientSession


async def fetch(url, session):
    async with session.get(url) as response:
        data = await response.read()
        print(data)
        return data


async def bound_fetch(sem, url, session):
    async with sem:
        await fetch(url, session)


async def run(r):
    url = "http://localhost:8080"
    tasks = []
    sem = asyncio.Semaphore(1000)

    async with ClientSession() as session:
        for i in range(r):
            task = asyncio.ensure_future(bound_fetch(sem, url, session))
            tasks.append(task)

        responses = asyncio.gather(*tasks)
        await responses
        print(responses.result())

number = 10
loop = asyncio.get_event_loop()

future = asyncio.ensure_future(run(number))
loop.run_until_complete(future)

Output of the code:

b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
b'<!DOCTYPE html>\n<html>\n<body>\n\n<h1>Hello World</h1>\n\n</body>\n</html>'
[None, None, None, None, None, None, None, None, None, None]

My question is why does responses not contain the actual response of the requests? Instead it is providing me with 'None' when actually I'm getting the expected response back during fetch() ?

Solution

  • Your bound_fectch co-routine, used as an intermediary step to use the semaphore, does not return an explicit result (and therefore, implicitly, returns None).

    Just change it to:

    async def bound_fetch(sem, url, session):
    async with sem:
        return await fetch(url, session)

    (note the "return" keyword. There is no matter in it being inside the with block: exiting the function exists the block, nonetheless)