Wait till all tasks are run in Celery python

Question

I am using celery in python for asynchronous tasks. I want to capture the result of it after all the tasks assigned to all workers are done.

For that, I am using the .get() method, but the problem with get() is that all the tasks are being assigned to a single worker which is synchronous but I want the tasks to be distributed to all the workers available.

Below is my snippet.

for url in urls:
    res = good_bad_urls.delay(url[1])
    res.get()
return JsonResponse(some_data)

Is there any other method in celery to wait until all the tasks run asynchronously?

Answer 1

but the problem with get() is that all the tasks are being assigned to a single worker which is synchronous

Well, not exactly. The task distribution works just the same (even if it can seem to do otherwise), and the tasks themselves are still async. The difference is that result.get() IS a blocking call - so in your case, it waits for the current task to finish until it launches the next one.

But anyway: the solution here is to use a Group. In your case, it should look something like

jobs = group([good_bad_urls.s(url[1]) for url in urls])
async_res = jobs.apply_async()  
result = async_res.get()

The get() call will now wait for all tasks to be finished, but they will be launched in parallel.