I want to introduce multiprocessing in my code using celery. But currently my queue implementation is in Kafka.
Currently celery website mentions of only these 4 brokers: http://docs.celeryproject.org/en/master/getting-started/brokers/index.html#broker-overview
Is it possible to integrate Celery with Kafka something similar to RabbitMQ mentioned below:
from celery import Celery
app = Celery('tasks', broker='pyamqp://guest@localhost//')
@app.task
def add(x, y):
return x + y
There was discussion on having kafka as broker for celery, And there is a partial implementation of it. I hope this helps you
https://github.com/celery/kombu/issues/301
Looks like it will be available this Oct 31 or so.