I have a system set up currently that is using celery with a redis backend to do a bunch of asynchronous tasks such as sending emails, pulling social data, crawling,etc. Everything is working great, but I am having group figuring out how to monitor the system (aka the number of queue up messages). I started looking through the celery source but I figured I would post my questions in here: First off, here are my configurations:
BROKER_BACKEND = "redis"
BROKER_HOST = "localhost"
BROKER_PORT = 6379
BROKER_VHOST = "1"
REDIS_CONNECT_RETRY = True
REDIS_HOST = "localhost"
REDIS_PORT = 6379
REDIS_DB = "0"
CELERY_SEND_EVENTS = True
CELERYD_LOG_LEVEL = 'INFO'
CELERY_RESULT_BACKEND = "redis"
CELERY_TASK_RESULT_EXPIRES = 25
CELERYD_CONCURRENCY = 8
CELERYD_MAX_TASKS_PER_CHILD = 10
CELERY_ALWAYS_EAGER =True
The first thing I am trying to do is monitor how many messages are in my queue. I assume, behind the scenes, the redis backend is just pushing/popping from a list, although I cannot seem to find that in the code. So I mock up a simulation where I start about 100 tasks and am trying to find them in redis: My celeryd is running like this: python manage.py celeryd -c 4 --loglevel=DEBUG -n XXXXX --logfile=logs/ celery.log So I should only have 4 concurrent workers at once ..... Two thing I do not understand: Problem 1: After I have queued up 100 task, and look for them on redis, I only see the following:
$ redis-cli
redis 127.0.0.1:6379> keys *
1) "_kombu.binding.celery"
redis 127.0.0.1:6379> select 1
OK
redis 127.0.0.1:6379[1]> keys *
1) "_kombu.binding.celery"
2) "_kombu.binding.celeryd.pidbox"
redis 127.0.0.1:6379[1]>
I cannot seem to find the tasks to get a number of how many are queued (technically, 96 should be since I only support 4 concurrent tasks)
Problem 2
$ ps aux | grep celeryd | cut -c 13-120
41258 0.2 0.2 2526232 9440 s004 S+ 2:27PM 0:07.35 python
manage.py celeryd -c 4 --loglevel=DEBU
41261 0.0 0.1 2458320 2468 s004 S+ 2:27PM 0:00.09 python
manage.py celeryd -c 4 --loglevel=DEBU
38457 0.0 0.8 2559848 34672 s004 T 12:34PM 0:18.59 python
manage.py celeryd -c 4 --loglevel=INFO
38449 0.0 0.9 2517244 36752 s004 T 12:34PM 0:35.72 python
manage.py celeryd -c 4 --loglevel=INFO
38443 0.0 0.2 2524136 6456 s004 T 12:34PM 0:10.15 python
manage.py celeryd -c 4 --loglevel=INFO
84542 0.0 0.0 2460112 4 s000 T 27Jan12 0:00.74 python
manage.py celeryd -c 4 --loglevel=INFO
84536 0.0 0.0 2506728 4 s000 T 27Jan12 0:00.51 python
manage.py celeryd -c 4 --loglevel=INFO
41485 0.0 0.0 2435120 564 s000 S+ 2:54PM 0:00.00 grep
celeryd
41264 0.0 0.1 2458320 2480 s004 S+ 2:27PM 0:00.09 python
manage.py celeryd -c 4 --loglevel=DEBU
41263 0.0 0.1 2458320 2480 s004 S+ 2:27PM 0:00.09 python
manage.py celeryd -c 4 --loglevel=DEBU
41262 0.0 0.1 2458320 2480 s004 S+ 2:27PM 0:00.09 python
manage.py celeryd -c 4 --loglevel=DEBU
If anyone could explain this for me, it would be great.
Your configuration has CELERY_ALWAYS_EAGER = True. This means that the tasks run locally and hence you won't see them in Redis. From the docs: http://celery.readthedocs.org/en/latest/configuration.html#celery-always-eager
CELERY_ALWAYS_EAGER
If this is True, all tasks will be executed locally by blocking until the task returns. apply_async() and Task.delay() will return an EagerResult instance, which emulates the API and behavior of AsyncResult, except the result is already evaluated.
That is, tasks will be executed locally instead of being sent to the queue.