I updated my Django project (from celery==2.5.5 and django-celery==2.5.5) to celery==3.0.7 and django-celery==3.0.6. And then strange things happend.
The first thing I noticed was that every date/time in Django Celery Admin interface was offset with 3 hours. Because my TIME_ZONE = 'Europe/Sofia' which right now is UTC/GMT +3 hours I think that the problem is related to the timezones.
I looked celerycam logs and found this
UPDATE "djcelery_workerstate" SET "hostname" = 'three', "last_heartbeat" = '2012-09-18 17:57:49.000701+03:00' WHERE "djcelery_workerstate"."id" = 1 ; args=(u'three', u'2012-09-18 17:57:49.000701+03:00', 1)
last_heartbeat is incorrect it has to be 2012-09-18 14:57:49.000701+03:00 or 2012-09-18 11:57:49.000701+00:00
Maybe some function is expecting the datetime not to be timezone aware, but the datetime passed is actually timezone aware.
My thoughts are also confirmed by an error in CeleryCam log which only happens when a task is retried! The task has this @task(default_retry_delay = 30, max_retries = 60) decorator.
This is the error:
[2012-09-18 14:46:54,001: ERROR/MainProcess] Error in timer: ValueError('Not naive datetime (tzinfo is already set)',)
Traceback (most recent call last):
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 92, in apply_entry entry()
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 48, in __call__ return self.fun(*self.args, **self.kwargs)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/celery/utils/timer2.py", line 149, in _reschedules return fun(*args, **kwargs)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/celery/events/snapshot.py", line 71, in capture self.state.freeze_while(self.shutter, clear_after=self.clear_after)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/celery/events/state.py", line 225, in freeze_while return fun(*args, **kwargs)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/celery/events/snapshot.py", line 68, in shutter self.on_shutter(self.state)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/django/db/transaction.py", line 209, in inner return func(*args, **kwargs)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/djcelery/snapshot.py", line 139, in on_shutter self._autocommit(_handle_tasks)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/djcelery/snapshot.py", line 115, in _autocommit fun()
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/djcelery/snapshot.py", line 133, in _handle_tasks self.handle_task(task)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/djcelery/snapshot.py", line 74, in handle_task "eta": maybe_make_aware(maybe_iso8601(task.eta)),
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/djcelery/utils.py", line 74, in maybe_make_aware return make_aware(value)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/djcelery/utils.py", line 52, in make_aware value = timezone.make_aware(value, timezone.utc)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/django/utils/timezone.py", line 269, in make_aware return timezone.localize(value, is_dst=None)
File "/opt/pythonenv/TICKETS/local/lib/python2.7/site-packages/pytz/__init__.py", line 231, in localize raise ValueError('Not naive datetime (tzinfo is already set)')
ValueError: Not naive datetime (tzinfo is already set)
After Django/Celery update I deleted all database tables and made syncdb
Timezone settings for my Django projects are:
TIME_ZONE = 'Europe/Sofia'
USE_TZ = True
Django/Celery settings are:
# RabbitMQ Broker
BROKER_HOST = 'localhost'
BROKER_PORT = 5672
BROKER_VHOST = 'tixgate'
BROKER_USER = 'tixgate'
BROKER_PASSWORD = '*******'
# Redis Result Backend
CELERY_RESULT_BACKEND = 'redis'
CELERY_REDIS_HOST = 'localhost'
CELERY_REDIS_PORT = 6379
CELERY_REDIS_DB = 4
CELERY_REDIS_PASSWORD = '*******'
CELERY_SEND_EVENTS = True
CELERY_TASK_RESULT_EXPIRES = 60 * 60 * 24 # 1 day
CELERY_ALWAYS_EAGER = False
and I don't set CELERY_ENABLE_UTC or CELERY_TIMEZONE explicitly
Edit: The problem persist with celery==3.0.9 and django-celery==3.0.9
Edit: And by the way celeryev shows the correct datetimes (in GMT I assume)
Edit: I use PostgreSQL database
Seems this should be fixed in new release and as a temporary workaround branch 3.0 was suggested https://github.com/celery/django-celery/issues/183#issuecomment-10846821.