Search code examples
pythonasync-awaitconcurrencypython-asyncio

Dynamically add new tasks after gather(*tasks)


For example this code:

async def f1(num):
    while True:
        print(num)
        await asyncio.sleep(2)

class ExampleClass:
  def __init__():
     self.tasks = []

  async def main():
    for i in range(10):
        tasks.append(asyncio.create_task(f1(i)))
    await asyncio.gather(*tasks)

  def add_new_task(task):
      self.tasks.append(task)

Then somewhere outside I call

ExampleClass.add_new_task(task)

What I need is to add new tasks and execute them asynchronously with the existing ones.

May be I should use any other constructions to implement what i want? What is important is that my tasks probably need to execute forever(forever polling)


Solution

  • So, I come to possible solution for my case with the idea of Louis. The approximate idea is to use fake async linfinite loop where all my task are executed and where I update new tasks within asyncio.TaskGroup :

       async def fake_generator():
            while True:
                yield None
    
        fake_gen = fake_generator()
        async with asyncio.TaskGroup() as tg:
    
            async for _ in fake_gen:
                await add_new_data()
    
                for element in self.data:
                    if not self.data[element]['in_use']:
                        tg.create_task(job(element))
                        self.data[element]['in_use'] = True
    
                await asyncio.sleep(60 * 60)