I've been setting up a test instance of django-celery and going through some of the basic celery examples and came across something that seems odd.
I first went through the "First steps with Django" celery page here: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
Everything worked correctly, as did the basic examples in the first few sections of the standard celery tutorial.
Once I started trying out some of the canvas primitives in the Django shell I get a NameError, as follows:
$ python manage.py shell
In [1]: from celerytest.tasks import add
In [2]: from celery import group
In [3]: group(add.s(i, i) for i in xrange(10))().get()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/django/core/management/commands/shell.pyc in <module>()
----> 1 group(add.s(i, i) for i in xrange(10))().get()
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/celery/canvas.pyc in __call__(self, *partial_args, **options)
397
398 def __call__(self, *partial_args, **options):
--> 399 tasks = [task.clone() for task in self.tasks]
400 if not tasks:
401 return
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/celery/utils/functional.pyc in __iter__(self)
286
287 def __iter__(self): # needed for Python 2.5
--> 288 return iter(self.data)
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/kombu/utils/__init__.pyc in __get__(self, obj, type)
292 return obj.__dict__[self.__name__]
293 except KeyError:
--> 294 value = obj.__dict__[self.__name__] = self.__get(obj)
295 return value
296
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/celery/utils/functional.pyc in data(self)
283 @cached_property
284 def data(self):
--> 285 return list(self.__it)
286
287 def __iter__(self): # needed for Python 2.5
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/django/core/management/commands/shell.pyc in <genexpr>((i,))
----> 1 group(add.s(i, i) for i in xrange(10))().get()
NameError: global name 'add' is not defined
But this next line works:
In [4]: add.delay(2,2).get()
Out[4]: 4
As does explicitly listing subtasks:
In [5]: group([add.s(1,2), add.s(3,4)])().get()
Out[5]: [3, 7]
If I use a regular Python shell and manually import settings instead of using manage.py shell, everything works. I.e.
$ ipython
In [1]: from testproj import settings
In [2]: from celerytest.tasks import add
In [3]: from celery import group
In [4]: group(add.s(i, i) for i in xrange(10))().get()
Out[4]: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
I've looked at the failing case with epdb and can't figure out exactly what's going on.
I've Googled around for this, and the only 100% similar reference I found was this issue on celery's github page: https://github.com/celery/celery/issues/1222. The poster mentions the same issues I'm having, but only says his "Python/Django shell is doing something strange" and doesn't elaborate further. Any ideas on what causes this or if there's somewhere else I should be looking?
After a significant amount of digging, I tracked down the source of the problem and it's actually already fixed on the Django trunk (it's not in the current 1.5 release, though, which is what I'm using).
It turns out that Django's shell command starts up iPython like this (in django/core/management/commands/shell.py)
from IPython import embed
embed()
However, calling embed() inside of a function causes iPython to start with separate local and global namespaces (iPython Github Issue 62). So lambda functions and generator expressions (like group) fail.
To fix this, iPython provides a different way of starting the shell that doesn't have this side effect.
from IPython.frontend.terminal.ipapp import TerminalIPythonApp
app = TerminalIPythonApp.instance()
app.initialize(argv=[])
app.start()
This should show up in an upcoming version (Django Issue). I tested with this patch applied to 1.5.1 and my code from the original works correctly when it's applied.