Search code examples
pythonpython-2.7rabbitmqpikaeventlet

RabbitMQ/Pika blocking consumer with evenlet


I am working on a larger application that requires eventlet, and now requires rabbitMQ as well. It appears that eventlet is causing the pika consumer thread to block execution of additional workers. I know Pika is not considered thread-safe, so I have everything, including the connection, within it's own thread. I would assume the blocking connection should only block the consumer thread. How can I get pika and eventlet to work together? In the below example, the workers thread never prints anything out, but commenting out eventlet.monkey_patch() allows both threads to execute.

import threading
import pika

import eventlet
eventlet.monkey_patch()


def callback(ch, method, properties, body):
    print body
    ch.basic_ack(delivery_tag=method.delivery_tag)


def consumer():
    connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
    channel = connection.channel()
    channel.queue_declare(queue='test', durable=True,
                          exclusive=False, auto_delete=False)
    channel.basic_consume(callback, queue='test')
    channel.start_consuming()


def start_consumer_thread():
    # initialize a listener thread
    consumer_thread = threading.Thread(target=consumer)
    consumer_thread.start()

def worker():
    start_consumer_thread()
    for x in range(1,10000):
        print x


x = threading.Thread(target=worker())
x.start()

Solution

  • Pika and eventlet.monkey_patch are not compatible. You will have to use eventlet without patching system calls, if that is possible.