On idle, the CPU usage of Celery docker container is extremely high with docker stats shows up to 600% usage. This behaviour is observed immediately after starting up the containers before any tasks or requests have been sent to Celery.
This is my run command in the celery docker file
CMD celery -A make_celery worker --pool=gevent --loglevel INFO
This deployment is on a Google Cloud VM with 2vCPU and instance CPU usage fluctuates between 80 and 90%
I have also tried with prefork workers but the behaviour is the same. Also tried to force the celery worker count to 1 and also run the container standalone without the other services.
Found the cause of the high CPU usage by enabling DEBUG logging on the celery instance.
CMD celery -A make_celery worker --loglevel DEBUG
There were an unusual number of ping calls which lead to checking the healthcheck configured in docker compose.
[DEBUG/MainProcess] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '********'} ticket:********]
Inside my docker compose I had
celery_worker:
build:
context: .
dockerfile: ./docker/prod/celery.Dockerfile
image: [repo/image]
healthcheck:
test: celery -A make_celery inspect ping
interval: 1s <-------------
timeout: 5s
retries: 10
env_file:
- .env
depends_on:
- redis
networks:
- app-network
restart: unless-stopped
Due to how the celery health check works, it creates new worker to run the ping command. However due to the 1 second interval it would keep creating a new worker immediately consecutively.
Changing this to a less frequent interval immediately reduced the CPU load.
celery_worker:
build:
context: .
dockerfile: ./docker/prod/celery.Dockerfile
image: [repo/image]
healthcheck:
test: celery -A make_celery inspect ping
interval: 60s
timeout: 5s
retries: 5
env_file:
- .env
depends_on:
- redis
networks:
- app-network
restart: unless-stopped