My celery beat docker container has (from what I believe) problems connecting the message broker.
I have a RabbitMQ container running:
....
rabbitmq:
image: rabbitmq:3.11.13-management-alpine
container_name: 'rabbitmq'
ports:
- 5672:5672
- 15672:15672
env_file:
- .env
volumes:
- ./data:/var/lib/rabbitmq/mnesia
networks:
- rabbitmq_go_net
....
And a Python container running a Django application with Celery (there are multiple more containers running the backend itself, celery workers, celery beat and celery flower, which all utilize the same backend files but handle different tasks):
....
celery-beat:
build:
context: .
dockerfile: ./compose/local/django/Dockerfile
container_name: 'beat'
image: beat
command: ["./wait-for-it.sh", "0.0.0.0:1337", "--", "/start-celerybeat"]
volumes:
- .:/app
env_file:
- .env
depends_on:
- rabbitmq
- backend
restart: always
....
Starting script for celery beat start.sh (executed by /start-celerybeat):
#!/bin/bash
set -o errexit
set -o nounset
rm -f './celerybeat.pid'
celery -A app beat -l INFO
app/celery.py:
from __future__ import absolute_import
import os
from celery import Celery
from celery.schedules import crontab
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'app.settings')
app = Celery('app')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()#(lambda: settings.INSTALLED_APPS)
@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
....
app/settings.py
INSTALLED_APPS = [
....
'django_celery_results',
'django_celery_beat',
....
]
CELERY_BROKER_URL = CONFIG.get('CELERY_BROKER_URL')
CELERY_RESULT_BACKEND = CONFIG.get('CELERY_RESULT_BACKEND')
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_BEAT_SCHEDULER = "django_celery_beat.schedulers:DatabaseScheduler"
.env
RABBITMQ_USER=user
RABBITMQ_PASSWORD=password
RABBITMQ_PORT=5672
RABBITMQ_HOST=rabbitmq
CELERY_BROKER_URL=amqp://${RABBITMQ_USER}:${RABBITMQ_PASSWORD}@localhost:${RABBITMQ_PORT}/${RABBITMQ_HOST}
CELERY_RESULT_BACKEND=amqp://
I receive the following error when starting up my docker-compose:
beat | [2024-02-20 15:19:54,198: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 6.0 seconds...
beat | [2024-02-20 15:20:00,212: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 8.0 seconds...
beat | [2024-02-20 15:20:08,223: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 10.0 seconds...
beat | [2024-02-20 15:20:18,235: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 12.0 seconds...
beat | [2024-02-20 15:20:30,248: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 14.0 seconds...
beat | [2024-02-20 15:20:44,265: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 16.0 seconds...
beat | [2024-02-20 15:21:00,283: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 18.0 seconds...
beat | [2024-02-20 15:21:18,307: ERROR/MainProcess] beat: Connection error: [Errno 111] Connection refused. Trying again in 20.0 seconds...
If you need more information than that, feel free to ask what more you need specifically.
Your CELERY_BROKER_URL is wrong. In Docker localhost means local loopback network interface within the same docker container.
Try this instead:
CELERY_BROKER_URL=amqp://${RABBITMQ_USER}:${RABBITMQ_PASSWORD}@rabbitmq:5672/
For more information you can check this post: From inside of a Docker container, how do I connect to the localhost of the machine?