Tornado celery integration hacks

Question

Since nobody provided a solution to this post plus the fact that I desperately need a workaround, here is my situation and some abstract solutions/ideas for debate.

My stack:

1. Tornado
2. Celery
3. MongoDB
4. Redis
5. RabbitMQ

My problem: Find a way for Tornado to dispatch a celery task ( solved ) and then asynchronously gather the result ( any ideas? ).

Scenario 1: (request/response hack plus webhook)

• Tornado receives a (user)request, then saves in local memory (or in Redis) a { jobID : (user)request} to remember where to propagate the response, and fires a celery task with jobID
• When celery completes the task, it performs a webhook at some url and tells tornado that this jobID has finished ( plus the results )
• Tornado retrieves the (user)request and forwards a response to the (user)

Can this happen? Does it have any logic?

Scenario 2: (tornado plus long-polling)

• Tornado dispatches the celery task and returns some primary json data to the client (jQuery)
• jQuery does some long-polling upon receipt of the primary json, say, every x microseconds, and tornado replies according to some database flag. When the celery task completes, this database flag is set to True, then jQuery "loop" is finished.

Is this efficient?

Any other ideas/schemas?

Answer 1

I stumbled upon this question and hitting the results backend repeatedly did not look optimal to me. So I implemented a Mixin similar to your Scenario 1 using Unix Sockets.

It notifies Tornado as soon as the task finishes (to be accurate, as soon as next task in chain runs) and only hits results backend once. Here is the link.