I'm trying to better understand the RabbitListener behavior when the broker goes down & when its brought back up.
I have setup a rabbit in a docker container over localhost:6672 and created a simple spring boot application to listen for messages. Also configured application.yml to point to this rabbit.
@RabbitListener(containerFactory = "myListenerContainerFactory",
bindings = @QueueBinding(value = @Queue(value = "MY_QUEUE"), exchange = @Exchange(value = "MY_EXCHANGE"), key = "MY_KEY"))
public void onMessage(final Mesg message) {
//some handling
}
Also defined the below container factory
@Bean
public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory cf) {
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(cf);
factory.setMessageConverter(new Jackson2JsonMessageConverter());
factory.setDefaultRequeueRejected(false);
factory.setConsumerTagStrategy(q -> "My App");
return factory;
}
I brought the spring boot app up when the rabbit container was running & it was able to auto create the exchange/queue & able to consume messages that i posted through rabbit admin console. I removed the rabbit container forcefully (docker rm my_instance -f) while the spring boot app was running and it started printing the below message.
2020-05-26 17:03:15.097 ERROR 26816 --- [ 127.0.0.1:6672] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: connection error
2020-05-26 17:03:15.614 INFO 26816 --- [cTaskExecutor-1] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer@69176296: tags=[[My App]], channel=Cached Rabbit Channel: AMQChannel(amqp://[email protected]:6672/,1), conn: Proxy@522c56e8 Shared Rabbit Connection: SimpleConnection@7578dfd0 [delegate=amqp://[email protected]:6672/, localPort= 65342], acknowledgeMode=AUTO local queue size=0
2020-05-26 17:03:15.618 INFO 26816 --- [cTaskExecutor-2] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: localhost:6672
2020-05-26 17:03:22.662 WARN 26816 --- [cTaskExecutor-2] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
2020-05-26 17:03:22.663 INFO 26816 --- [cTaskExecutor-2] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer@31978e0c: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2020-05-26 17:03:22.663 INFO 26816 --- [cTaskExecutor-3] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: localhost:6672
2020-05-26 17:03:29.683 WARN 26816 --- [cTaskExecutor-3] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
After starting the rabbit container again, spring boot app seems to have detected it however it printed the below message & i don't think it has established a successful connection. Looking at the logs it seems to have 3 retries setup by default, is there a config that i can use to increase the limit?
Is there any best practice around handling such cases? My app is expected to connect with different rabbit clusters and its possible that 1 or the other may go on maintenance & i needed a way to re-connect automatically as soon as the rabbit cluster is back up.
2020-05-26 17:05:18.303 INFO 26816 --- [TaskExecutor-19] o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer@5053bc3c: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2020-05-26 17:05:18.306 INFO 26816 --- [TaskExecutor-20] o.s.a.r.c.CachingConnectionFactory : Attempting to connect to: localhost:6672
2020-05-26 17:05:18.348 INFO 26816 --- [TaskExecutor-20] o.s.a.r.c.CachingConnectionFactory : Created new connection: myConnectionFactory#ebfe707:19/SimpleConnection@19d03268 [delegate=amqp://[email protected]:6672/, localPort= 49260]
2020-05-26 17:05:18.396 WARN 26816 --- [TaskExecutor-20] o.s.a.r.listener.BlockingQueueConsumer : Failed to declare queue: MY_QUEUE
2020-05-26 17:05:18.402 WARN 26816 --- [TaskExecutor-20] o.s.a.r.listener.BlockingQueueConsumer : Queue declaration failed; retries left=3
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[MY_QUEUE]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:710) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.passiveDeclarations(BlockingQueueConsumer.java:594) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:581) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1196) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1041) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.io.IOException: null
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:126) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:122) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:144) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:1006) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:52) ~[amqp-client-5.4.3.jar:5.4.3]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1110) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at com.sun.proxy.$Proxy283.queueDeclarePassive(Unknown Source) ~[na:na]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:689) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
... 5 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'MY_QUEUE' in vhost '/', class-id=50, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:494) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:288) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:138) ~[amqp-client-5.4.3.jar:5.4.3]
... 14 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'MY_QUEUE' in vhost '/', class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:516) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:670) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48) ~[amqp-client-5.4.3.jar:5.4.3]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:597) ~[amqp-client-5.4.3.jar:5.4.3]
... 1 common frames omitted
2020-05-26 17:05:23.420 WARN 26816 --- [TaskExecutor-20] o.s.a.r.listener.BlockingQueueConsumer : Failed to declare queue: MY_QUEUE
2020-05-26 17:05:23.425 WARN 26816 --- [TaskExecutor-20] o.s.a.r.listener.BlockingQueueConsumer : Queue declaration failed; retries left=2
reply-text=NOT_FOUND - no queue 'MY_QUEUE' in vhost '/'
You have a MY_QUEUE which is declared as auto-delete and looks like there is no its recreation during connection establishing. Consider to have it as a bean and let RabbitAdmin to handle its creation on the reconnection.
See more info in docs: https://docs.spring.io/spring-amqp/docs/2.2.7.RELEASE/reference/html/#broker-configuration