Search code examples
pythonasync-awaitpython-asynciogenerator

asyncio.timeout for async generator does not work


Why does asyncio.timeout not work when wrapping async generator like in example below?

async def generate_many_numbers():
    for i in range(1000000000):
        yield i

async def main2():
    async with asyncio.timeout(1):
        async for number in generate_many_numbers():
            print(number)

asyncio.run(main2())

Can it be the same issue which is described here? https://peps.python.org/pep-0789/

I have noticed that adding async dummy sleep helps...

async def generate_many_numbers():
    for i in range(1000000000):
        await asyncio.sleep(0)
        yield i

async def iterate_through_numbers():
    async for number in generate_many_numbers():
        print(number)

async def main2():
    async with asyncio.timeout(1):
        async for number in generate_many_numbers():
            print(number)

async def main3():
    try:
        await asyncio.wait_for(iterate_through_numbers(), timeout=1)
    except asyncio.TimeoutError:
        print("The task timed out!")

Solution

  • The function generate_many_numbers never returns control to the asyncio event loop, so asyncio.timeout has no chance to cancel it before it finishes.

    To return control to the event loop, there must be (in the simplest case) an await statement. If there is no other reason to use await, you can use await asyncio.sleep(0), see e.g. Why does asyncio.sleep(0) make my code faster?

    I don't think this has to do with PEP 789.