How to run a list of task in by X a time? And wait for each group to finish

Question

I have a group of 1000 tasks.

import asyncio

async def my_task(x):
    await asyncio.sleep(0.1)
    print(f"done: {x}")


async def main():
    my_tasks = []
    for x in range(1000):
        my_tasks.append(lambda: my_task)         

    # ???
    # how to scoop up the consequent 10 out of `my_tasks`
    # to execute them asyncronously?
    # and then wait for them?
    # 

    # ??
    # asyncio.create_task(my_task())
    # pending = asyncio.all_tasks()
    # group = asyncio.gather(*pending, return_exceptions=True)
    # await group

I want to run them 10 by 10. That is, 10 ones at a time. Then wait for them (10) to finish, and only after this do run another 10 ones, and so on.

How to do it?

Answer 1

You can achieve this by using asyncio.gather to run the tasks concurrently in groups of 10. Here's an example of how you can modify your main function to accomplish this:

import asyncio

async def my_task(x):
    await asyncio.sleep(0.1)
    print(f"done: {x}")

async def main():
    my_tasks = [my_task(x) for x in range(1000)]

    # Run tasks in groups of 10
    for i in range(0, len(my_tasks), 10):
        group = my_tasks[i:i+10]
        await asyncio.gather(*group)

asyncio.run(main())

This code creates a list of 1000 tasks and then iterates through it in steps of 10. For each group of 10 tasks, it uses asyncio.gather to run them concurrently and waits for them to finish before moving on to the next group.

Note: In some python code editors (such as google colab notebooks) you should use await main() instead of asyncio.run(main())