I'm trying to setup rabbitmq/celery/django-celery/django so that it is "rebootproof", i.e. it all just comes back up by itself. Everything seems to work fine except this:
When I reboot, all services get started, but it seems celeryd is started before rabbitmq, and celerybeat gets subsequently terminated because it can't connect (?):
[2011-06-14 00:48:35,128: WARNING/MainProcess] celery@inquire has started.
[2011-06-14 00:48:35,130: INFO/Beat] child process calling self.run()
[2011-06-14 00:48:35,131: INFO/Beat] Celerybeat: Starting...
[2011-06-14 00:48:35,134: ERROR/MainProcess] Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 2 seconds...
[2011-06-14 00:48:35,688: INFO/Beat] process shutting down
[2011-06-14 00:48:35,689: WARNING/Beat] Process Beat:
[2011-06-14 00:48:35,689: WARNING/Beat] Traceback (most recent call last):
...
[2011-06-14 00:48:35,756: WARNING/Beat] File "/home/inquire/inquire.env/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 220, in create_transport
[2011-06-14 00:48:35,760: WARNING/Beat] return TCPTransport(host, connect_timeout)
[2011-06-14 00:48:35,761: WARNING/Beat] File "/home/inquire/inquire.env/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 58, in __init__
[2011-06-14 00:48:35,761: WARNING/Beat] self.sock.connect((host, port))
[2011-06-14 00:48:35,761: WARNING/Beat] File "<string>", line 1, in connect
[2011-06-14 00:48:35,761: WARNING/Beat] error: [Errno 111] Connection refused
[2011-06-14 00:48:35,761: INFO/Beat] process exiting with exitcode 1
[2011-06-14 00:48:37,137: ERROR/MainProcess] Consumer: Connection Error: [Errno 111] Connection refused. Trying again in 4 seconds...
On Ubuntu, I installed rabbitmq-server with apt, django-celery with pip into my virtualenv, then I symlinked the "celeryd" initscript I got from https://github.com/ask/celery/tree/master/contrib/debian/init.d in /etc/init.d, configured it in /etc/default/celeryd to use the django celeryd from my virtualenv, and made it "rebootproof" via (maybe "defaults" is the problem?)
update-rc.d celeryd defaults
Rather than running celeryd and celerybeat with separate initscripts, I just configured celeryd to include Beat (maybe that's the problem?):
CELERYD_OPTS="-v 2 -B -s celery -E"
Any pointers how to solve this issue?
If I
sudo /etc/init.d/celeryd restart
there are no complaints:
[2011-06-14 00:54:29,157: WARNING/MainProcess] celery@inquire has started.
[2011-06-14 00:54:29,161: INFO/Beat] child process calling self.run()
[2011-06-14 00:54:29,162: INFO/Beat] Celerybeat: Starting...
but I need to eliminate the need for any manual steps.
celerybeat's dependency on the broker service was indeed the issue. Rather than installing the initscript with
update-rc.d celeryd defaults
with the rabbitmq-server script being installed as sequence number 20 for start and kill, celerybeat's dependency must be resolved by explicitly starting it after (and killing it before) rabbitmq-server by using
update-rc.d celeryd defaults 21 19
NB: I've actually opted for the separate celerybeat service instead of the -B invocation, and only did 21 19 for that script, i.e. the one with the problem.