What is the proper way to start bunch of coroutines in loop using python asyncio.Semaphore?

Question

I want to make a function, which will start bunch of coroutines, and wait for all of them to get done, by using asyncio Semaphore class. After processing this bunch of coroutines, I want to setup a timeout, to wait until running the next bunch.

I had tried to make this by using two for loops, but I'm not quite sure, that this is the best way to solve this problem. Here is the code:

async def start(self):
        sem = asyncio.Semaphore(10)
        latest_block_number = await self.w3.eth.block_number
        tasks = []
        for i in range(latest_block_number, latest_block_number-self.blocks_needed, -10):
            for j in range(i, i - 10, -1):
                task = asyncio.create_task(self.get_transactions_of_block(current_block_number=j, sem=sem))
                tasks.append(task)

            await asyncio.gather(*tasks)
            tasks.clear()
            await asyncio.sleep(60)

Can you please help me with your ideas? Thank you!

Answer 1

you can do this with asyncio.gather(). This will bunch everything together (or in technical terms, allows you to run multiple coroutine functions concurrently using asynchronous I/O) and ensure that they are run. Then you can invoke a synchronous sleep.

Also, there is a link to the docs: https://docs.python.org/3/library/asyncio-task.html

This is some code i did a while ago to test, but i have added a for loop that also waits 5 seconds between each loop.

import asyncio
import time


def main():
    ''' main entry point for the program '''
    # create the event loop and add to the loop
    # or run directly.

    asyncio.run(main_async())
    return

async def main_async():
    ''' the main async function '''
    # await foo()
    # await bar()
    for i in range(3):
        await asyncio.gather(foo(), bar())
        print('wait a bit..')
        time.sleep(5)
    return

async def foo():
    print('before the sleep')
    await asyncio.sleep(2)
    # time.sleep(2)
    print('world')
    return

async def bar():
    print('hello')
    await asyncio.sleep(0)
    return



if __name__=='__main__':
    ''' This is executed when run from the command line '''
    main()

The process above returns this:

hello
world
wait a bit..
before the sleep
hello
world
wait a bit..
before the sleep
hello
world
wait a bit..