Search code examples
celerycelerybeat

Add a delay between subtasks of a group

I am creating a Celery group to execute a bunch of processes in parallel. The following is the code -

@periodic_task(run_every=timedelta(minutes=1))
def schedule_pivots():
    today_list="A,B,C,D"
    g = group(get_pivots.s(l,'OPT') for l in today_list.split(','))
    g.set(countdown=100).delay()

'get_pivots' is the task called parallely. This task calls an external API which rate-limits if called within 500ms. Is there a way for me to introduce a delay between each execution of the task 'get_pivots' ?

Solution

  • Just tested this

    @celery.task(rate_limit='10/m')
def add(x, y):
    return x + y

    While calling this task from the python interpreter

    >>> g = group(add.s(i, i + 1) for i in range(10))
>>> r = g()
>>> r.get()

    The last part freezes till the everything is done, Hope that helps