we're currently prototyping a microservice (Node v8.3.0) which have to consume about 60-150 messages per second of a RabbitMQ (RabbitMQ 3.6.12, Erlang 19.2.1). Sometimes it works like a charm and there are no remaining messages in the queue. But most of the time the messages stuck, only 5-20 messages per second get handled and and accumulate to up to 3M messages in the queue.
Now we're really curious about how to handle all those messages with a single consumer. Because there are already some Java consumer handling all those messages without any delays. We use this node library based on amqplib. Furthermore the handler acknowledges the incoming messages immediately — the business logic is absolutely asynchronous. So even without any business logic it stucks. The exchange's type is topic, it isn't durable and the queue has the auto-deleting feature enabled. We tried disabled prefetch, prefetch = 1 and 100, without any success.
So..
1) Which AMQP/RabbitMQ library for node do you use?
2) How many messages get handled per second?
3) Any further improvements/suggestions?
Thanks!
The fact that you have a Java consumer that works correctly points to either amqplib-easy, amqplib or your code as the culprit. Also, note that using a single queue in RabbitMQ is an anti-pattern as queues are the unit of concurrency in the broker.
I have put together a test project that includes a README on running a Node consumer with the RabbitMQ PerfTest application (Java). You should familiarize yourself with PerfTest as it provides many features for evaluating the performance capabilities of your environment.
In my test environment, I can sustain a publish rate of 4096msg/sec easily. If I increase that to 8192 I can see messages back up due to the fact that the Node app can't consume fast enough. It would be interesting to compare using "plain" amqplib as well.