Search code examples
pythondjangorabbitmqcelerysystem-design

RabbitMQ : How to check queue content when processing a task using celery?

I have setup a basic message and task queue using RabbitMQ and Celery in my Django app. According to my understanding the when I use delay menthod, that pushes my tasks to the rabbitMQ queue and one of my workers from my celery app fetches the same from queue to execute the same. Is there any way in which when I push the task to the queue using delay I can view the same on my message queue RabbitMQ management portal? Since the tasks are almost instantly fetched and processed there is no chance of viewing them on the portal.

This is what I tried which I think is wrong by adding a sleep timer in my task method.

sudo nano tasks.py

from __future__ import absolute_import, unicode_literals
from celery import shared_task
import time

@shared_task
def add(x, y):
    time.sleep(100)
    return x + y

Started my celery app

(myprojectenv) root@ubuntu-s-1vcpu-1gb-blr1-01:/etc/myproject# celery -A myproject worker -l info

Pushed the tasks for processing

(myprojectenv) root@ubuntu-s-1vcpu-1gb-blr1-01:/etc/myproject# python3 manage.py shell
Python 3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from app1.tasks import add
>>> add.delay(1,2)

On celery window

[2022-06-10 06:16:15,182: INFO/MainProcess] celery@ubuntu-s-1vcpu-1gb-blr1-01 ready.
[2022-06-10 06:17:38,485: INFO/MainProcess] Task app1.tasks.add[be566921-b320-466c-b406-7a6ed7ab06e7] received

But I don't see the content in my queue

enter image description here

Solution

  • Disclaimer: I'm not sure how perfect this way is - But, it will do the job.

    • Step 1: Turn off the worker (that is kill/terminate the process initiated by celery -A myproject worker command)
    • Step 2: Push the messages into RabbitMQ (Just call the task from Django shell with .delay() method)
    • Step 3: Check the RabbitMQ management console.

    Note: You can consume/view/pop the messages from the queue. Depending on the pop mechanism you choose (ack or nack -(correct me if I'm wrong)), the worker will be able to consume the messages once it's live.