How should a Node.js microservice survive a Rabbitmq restart?

Question

I've been working on an example of using Rabbitmq for communication between Node.js microservices and I'm trying to understand the best way for these microservices to survive a restart of the Rabbitmq server.

Full example is available on Github: https://github.com/ashleydavis/rabbit-messaging-example

You can start the system up by changing to the broadcast sub-directory and using docker-compose up --build.

With that running I open another terminal and issue the following command to terminate the Rabbit server docker-compose kill rabbit.

This causes a Node.js unhandled exception to kill my sender and receiver microservices that were connected to the Rabbitmq server.

Now I'd like to be able to restart the Rabbitmq server (using docker-compose up rabbit) and have the original microservices come back online.

This is intended to run under Docker-Compose for development and Kubernetes for production. I could just set this up so that the microservices restart when they are terminated by the disconnection from Rabbitmq, but I'd prefer it if the microservices could stay online (they might be doing other work that shouldn't be interrupted) and then reconnect to Rabbitmq automatically when it becomes available again.

Anyone know how to achieve automatic reconnection to Rabbitmq using the ampq library?

Answer 1

Just picking the sender service as an example on how to deal with it. The error that is causing node to exit is that here is no 'error' handler on the stream the writer users.

If you modify this part of the code. https://github.com/ashleydavis/rabbit-messaging-example/blob/master/broadcast/sender/src/index.js#L13

Change the line in sender/src/index.js from

const messagingConnection = await retry(() => amqp.connect(messagingHost), 10, 5000);

to

const messagingConnection = await retry(() => amqp.connect(messagingHost), 10, 5000)
    .then(x => {
        return x.on('error', (err) => {
            console.log('connect stream on error', err)
        });
    });

Just having the error handler means that the node process no longer exists with unhandled exception. This does not make the sender code correct, it now needs to be modified to know if it has a connection, only send data if it has a connection, retry to connect if it has no connection.

A similar fix for the receiver can be applied

This is a useful reference for when node requires setup to not exit. https://medium.com/dailyjs/how-to-prevent-your-node-js-process-from-crashing-5d40247b8ab2