From what I've read, it seems to be that a celery worker(s) stops, it should be able to consume from the queue upon start again.
To test this, I spun up 3 virtual machines, each machine having 2 celery workers. I am using a celery daemon. One machine is has the client script, and one is acting as a broker, though all of them are running rabbitmq.
When I start the client script, all my workers are getting tasks (from the logs I can tell). To test what happens upon failure, I issue
service celeryd stop
to one of my machines. Then after a minute, I issue
service celeryd start
to the machine I stopped a minute ago. However, upon issuing this command, my client script freezes and stalls, and my workers on the above machine never receive tasks again.
Any idea on what is going on? I'm guessing a possibility is that the last task the celery workers were working on before being shut down is somehow interfering?
FYI: I am deleting temporary queues once I have received the result in my client script, but this I don't think this would affect the above problem.
Yes, you are right. When shutdown is initiated the worker will finish all currently executing tasks before it actually terminates[refer to [celery document].
And you'd better to take a look at the log of the corresponding worker to make sure it is really the root issue.
By the way, you could use flower to monitor celery.