Problem with dockerizing a Django project with Celery and RabbitMQ

Question

I am trying to dockerize my Django project with Celery and RabbitMQ. I have a Docker Compose file, but I am having trouble getting it to work.

Here is my Docker Compose file:

services:
  rabbitmq:
    image: rabbitmq:latest
    networks:
      - main
    ports:
      - "5672:5672"
    restart: always
    environment:
      - RABBITMQ_DEFAULT_USER=guest
      - RABBITMQ_DEFAULT_PASS=guest

  celery_worker:
    image: python:3.10
    command: celery -A zwitter worker -l INFO
    build:
      context: .
    depends_on:
      - app
      - rabbitmq
    environment:
      - CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672/
    networks:
      - main
    restart: always

  redis:
    image: redis:latest
    ports:
      - "6379:6379"
    volumes:
      - cache:/data
    command: redis-server --save 60 1
    restart: always

  app:
    image: python:3.10
    build: .
    command: bash -c "python manage.py makemigrations && python manage.py migrate && python manage.py runserver 0.0.0.0:8000"
    volumes:
      - .:/src/
    ports:
      - "8000:8000"

networks:
  main:

volumes:
  cache:

When I try to start the Docker Compose containers, everything works fine and the containers are all up. But when any celery task is about to run in my django project, I get the following error:

[Errno -3] Temporary failure in name resolution

I have tried all of the following, but the problem still persists:

• Making sure that the RabbitMQ server is running and accessible.
• Making sure that the Celery worker is configured to connect to the correct RabbitMQ server.
• Checking the network connection between the Celery worker container and the RabbitMQ server.

Answer 1

For the two containers to connect to each other, they have to be on the same Docker network. Two of your containers (rabbitmq and celery_worker) are explicitly attached to networks: [main], but the other two (redis and app) aren't. For these two containers, Compose creates a network named default and attaches them to it.

The easiest solution here is to delete all of the networks: blocks from the file. Compose will attach all of the containers to the automatically-created default network. Since they'll all be on the same network, DNS resolution will work as expected.

(This is probably not your problem, but for the two containers you build: ., you should also delete the image: python:3.10 lines, and then manually docker pull python:3.10 once. If you have both build: and image: then this sets the name of the image that gets built, which means that your image builds are actually appending to the previous build and not starting from a clean Python base image.)