I have a fast api application with celery and rabbitmq running in docker (compose). Here is the compose configuration:
services:
scraper:
build:
context: fastapi-scrapers
dockerfile: Dockerfile
cpus: "0.7"
# ports:
# - '8000:8000'
environment:
- TIMEOUT="120"
- WEB_CONCURRENCY=2
networks:
- scrape-net
volumes:
- ../images/:/app/images/:rw
extra_hosts:
- "host.docker.internal:host-gateway"
flower:
image: mher/flower
ports:
- '5555:5555'
environment:
- CELERY_BROKER_URL=amqp://admin:pass@rabbitmq:5672/
# - CELERY_BROKER_URL=redis://redis:6379/0
- FLOWER_BASIC_AUTH=admin:pass
depends_on:
- scraper
networks:
- scrape-net
rabbitmq:
image: "rabbitmq:latest"
# ports:
# - '5672:5672'
# - "15672:15672"
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=pass
networks:
- scrape-net
extra_hosts:
- "host.docker.internal:host-gateway"
networks:
scrape-net:
driver: bridge
This is the Dockerfile for the fastapi application:
FROM python:3.9
WORKDIR /code
COPY ./requirements.txt /code/requirements.txt
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
COPY ./app /code/app
# CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
CMD ["bash", "-c", "celery -A app.celery.tasks worker --loglevel=info --concurrency=8 -E -P eventlet & uvicorn app.main:app --host 0.0.0.0 --port 8000"]
And here is the celery code in the app:
celery_app = Celery('tasks', broker='amqp://admin:pass@rabbitmq:5672/')
celery_app.conf.update(
CELERY_RESULT_EXPIRES=3600,
CELERY_AMQP_TASK_RESULT_EXPIRES=3600
)
Fast api application logs:
site-scraper-1 | INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
site-scraper-1 | INFO: Started parent process [8]
site-scraper-1 | INFO: Started server process [11]
site-scraper-1 | INFO: Waiting for application startup.
site-scraper-1 | INFO: Application startup complete.
site-scraper-1 | INFO: Started server process [10]
site-scraper-1 | INFO: Waiting for application startup.
site-scraper-1 | INFO: Application startup complete.
site-scraper-1 | /usr/local/lib/python3.9/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
site-scraper-1 | absolutely not recommended!
site-scraper-1 |
site-scraper-1 | Please specify a different user using the --uid option.
site-scraper-1 |
site-scraper-1 | User information: uid=0 euid=0 gid=0 egid=0
site-scraper-1 |
site-scraper-1 | warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
site-scraper-1 |
site-scraper-1 | -------------- celery@d4dec9482220 v5.3.6 (emerald-rush)
site-scraper-1 | --- ***** -----
site-scraper-1 | -- ******* ---- Linux-5.10.0-26-amd64-x86_64-with-glibc2.36 2024-03-10 22:26:52
site-scraper-1 | - *** --- * ---
site-scraper-1 | - ** ---------- [config]
site-scraper-1 | - ** ---------- .> app: tasks:0x7fd3a198f2e0
site-scraper-1 | - ** ---------- .> transport: amqp://admin:**@rabbitmq:5672//
site-scraper-1 | - ** ---------- .> results: disabled://
site-scraper-1 | - *** --- * --- .> concurrency: 8 (eventlet)
site-scraper-1 | -- ******* ---- .> task events: ON
site-scraper-1 | --- ***** -----
site-scraper-1 | -------------- [queues]
site-scraper-1 | .> celery exchange=celery(direct) key=celery
site-scraper-1 |
site-scraper-1 |
site-scraper-1 | [tasks]
site-scraper-1 | . app.celery.tasks.start_scrape
site-scraper-1 |
site-scraper-1 | [2024-03-10 22:26:52,909: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
site-scraper-1 | whether broker connection retries are made during startup in Celery 6.0 and above.
site-scraper-1 | If you wish to retain the existing behavior for retrying connections on startup,
site-scraper-1 | you should set broker_connection_retry_on_startup to True.
site-scraper-1 | warnings.warn(
site-scraper-1 |
site-scraper-1 | [2024-03-10 22:27:13,338: ERROR/MainProcess] consumer: Cannot connect to amqp://admin:**@rabbitmq:5672//: [Errno -3] Lookup timed out.
site-scraper-1 | Trying again in 2.00 seconds... (1/100)
site-scraper-1 |
site-scraper-1 | [2024-03-10 22:27:35,769: ERROR/MainProcess] consumer: Cannot connect to amqp://admin:**@rabbitmq:5672//: [Errno -3] Lookup timed out.
site-scraper-1 | Trying again in 4.00 seconds... (2/100)
This was working perfectly fine for months, and now is throwing this error in the fast api application. The rabbitmq logs seem to be fine, the user is created and the rabbitmq instance is started up as expected. The flower container is able to connect to the rabbitmq container without any issues. It is just the fast api container that is having issues.
I have tried using localhost in the amqp instead of rabbitmq, but that hasn't worked. It also does not work with docker.host.internal. I have tried updating the python app's dependencies to the latest versions, and that has not changed anything either. I have tried using Redis for the message broker, but it throws the same timeout error, so I think it has something to do with my python app.
Turned it was fixed in a later version of the python docker. So I updated:
FROM python:3.9
to
FROM python:latest
and now its working fine again.