RabbitMQ Client shuts down the MessageListener and cannot be recovered

Question

I use RabbitMQ AMQP 2.2.7. I have the RabbitMQ cluster running between 2 Spring boot application. One application posts some messages to the other one. It was running well for sometime but suddenly for the last few days, i see that MessageListener in the application that consumes the message goes down for some reason (May be the master server node went down).

   <dependency>
            <groupId>org.springframework.amqp</groupId>
            <artifactId>spring-rabbit</artifactId>
            <version>2.2.7.RELEASE</version>
   </dependency>

2020-07-22 00:26:33.007 ERROR 24 --- [tContainer#1-15] o.s.a.r.l.SimpleMessageListenerContainer : Stopping container from aborted consumer
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit@rad497159-mq-1.node.dc1.a9ssvc' of durable queue 'ORDER' in vhost '/' is down or inaccessible, class-id=50, method-id=10)
Caused by: java.io.IOException: null
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
2020-07-22 00:26:33.005 ERROR 24 --- [tContainer#1-15] o.s.a.r.l.SimpleMessageListenerContainer : Consumer threw missing queues exception, fatal=true
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1350) ~[spring-rabbit-2.2.7.RELEASE.jar:2.2.7.RELEASE]

@Configuration
public class MessageConfiguration {

    public static final String ORDER_QUEUE_NAME = "ORDER";

    public static final String EXCHANGE = "directExchange";

    @Bean
    Queue deadLetterQueue() {
        return QueueBuilder.durable(ORDER_QUEUE_NAME).build();
    }

    @Bean
    public Queue orderQueue(){
        return QueueBuilder.durable(ORDER_QUEUE_NAME)
                .build();
    }

    @Bean
    public DirectExchange directExchange(){
        return new DirectExchange(EXCHANGE,true,false);
    }

    @Bean
    public Binding firstBinding(Queue orderQueue, DirectExchange directExchange){
        return BindingBuilder.bind(orderQueue).to(directExchange).with(ORDER_QUEUE_NAME);
    }
}

@RabbitListener(queues = MessageConfiguration.ORDER_QUEUE_NAME)
public void receiveOrder(final String orderString) {
}

The problem is RabbitMQ message listener shuts down indefinitley and there is no other way to recover. Restarting the application solves the problem. So i would like have one of the following solution but dont know how to do that

1. Listen for the shutdown notification from SimpleMessageListener and restart the application
2. Make the application retry more times before shutting down the listener

Could someone please suggest some way ?

There is already the same issue in stackoverflow without solution (How to avoid shutdown of SimpleMessageListenerContainer in case of unexpected errors?)

Answer 1

home node 'rabbit@rad497159-mq-1.node.dc1.a9ssvc' of durable queue 'ORDER' in vhost '/' is down or inaccessible, class-id=50, method-id=10)

It looks like you are using a cluster but without using a mirrored (HA) queue.

In that case, if the queue's home node goes down, you will get that error.

Consumer threw missing queues exception, fatal=true

If you set the container property missingQueuesFatal to false, the container will keep trying until the node is back up.

By default, this is true and the container will stop after 3 attempts 5 seconds apart (by default).