Celery single task persistent data

Question

Lets say a single task is enough for a machine to stay very busy for a few minutes.

I want to get the result of the task, then depending on the result, have the worker perform the same task again.

The question I cannot find an answer to is this: Can I keep data in memory on the worker machine in order to use it on the next task?

Answer 1

Yes you can. The documentation (http://docs.celeryproject.org/en/latest/userguide/tasks.html#instantiation) is a bit vague and I'm not sure if this is the best way, but you can do something like this:

This is what you'd like to do, but doesn't work:

# This doesn't work
a = 0
@celery.task
def mytask(x):
    a += x
    return a

This is how to make it work:

from celery import Task, registry
@celery.task
class MyTask(Task):
    def __init__(self):
        self.a = 0

    def run(self, x):
        self.a += x
        return self.a
mytask = registry.tasks[MyTask.name]

According to the docs:

A task is not instantiated for every request, but is registered in the task registry as a global instance. This means that the __init__ constructor will only be called once per process, and that the task class is semantically closer to an Actor. ... If you have a task, ... And you route every request to the same process, then it will keep state between requests."

I've had success with this but I don't know if there are better methods.