Search code examples
javaspring-bootrabbitmqspring-cloud-streamspring-rabbit

RabbitMQ Binder null x-death within retry loop creating infinite retries before killing consumer


I have a retry loop going with use of the dead letter queue to schedule retries until hitting a limit. The issue I have is the x-death header is always coming back null whilst it's clearly aware of the failure as x-exception-message and x-exception-stacktrace are present.

This is what's happenning in my Spring tests where I use a testcontainers version of rabbit with 3.9-management docker image

Here is an example of Rabbit MQ binder class:

@EnableBinding(Sink.class)
@Service
@Slf4j
public class RabbitMqListener {
  
  @Value("${events.retries:10}")
  private long maxRetries;

  @StreamListener(target = Sink.INPUT)
  public void process(Event<Map<String, Object>> event,
                      @Header(name = "x-death", required = false) Map<?, ?> death,
                      @Headers Map<?,?> headers,
                      @Header(AmqpHeaders.DELIVERY_TAG) Long deliveryTag) {
    log.debug("Received event with Id:{} and deliveryTag:{}", event.getId(), deliveryTag);
    log.debug("headers:{}", headers);
    log.debug("x-death header:{}", death);
    if (death != null && death.get("count") != null &&
        Long.valueOf(death.get("count").toString()).compareTo(maxRetries) >= 0) {
      log.error("Max retries reached for event with Id:{}", event.getId());
      throw new ImmediateAcknowledgeAmqpException("Failed after " + (maxRetries + 1) + " attempts");
    }
    
    try {
      processEvent(event);
    } catch (Exception e) {
      throw new AmqpRejectAndDontRequeueException("Fail to evaluate event with id:" + event.getId(), e);
    }

  }

  private boolean processEvent(Event<Map<String, Object>> event) {
    //redacted - you can trust that in this test case it's throwing an exception
  }

}

My config:

spring:
  rabbitmq:
    host: localhost
    port: 5672
    username: guest
    password: guest
    dynamic: true
  cloud.stream:
    bindings:
      input:
        destination: events
        group: service
        binder: rabbit
        consumer:
          maxAttempts: 1
    rabbit:
      bindings:
        input:
          consumer:
            autoBindDlq: true
            dlqTtl: 1500
            deadLetterExchange:
    default.contentType: application/json

And some logs for the test:

[INFO] [main] 14:03:58.448 SpringBootMockServletContext - Initializing Spring TestDispatcherServlet ''
[INFO] [main] 14:03:58.448 TestDispatcherServlet - Initializing Servlet ''
[INFO] [main] 14:03:58.471 TestDispatcherServlet - Completed initialization in 23 ms
[INFO] [main] 14:03:58.577 CachingConnectionFactory - Attempting to connect to: [localhost:61215]
[INFO] [main] 14:03:58.584 CachingConnectionFactory - Created new connection: rabbitConnectionFactory.publisher#6b28d4e4:0/SimpleConnection@40941b54 [delegate=amqp://[email protected]:61215/, localPort= 61229]
[ERROR] [AMQP Connection 127.0.0.1:61215] 14:03:58.595 CachingConnectionFactory - Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)
[ERROR] [AMQP Connection 127.0.0.1:61215] 14:03:59.616 CachingConnectionFactory - Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)
[ERROR] [AMQP Connection 127.0.0.1:61215] 14:04:01.633 CachingConnectionFactory - Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)
[ERROR] [AMQP Connection 127.0.0.1:61215] 14:04:05.648 CachingConnectionFactory - Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)
[ERROR] [AMQP Connection 127.0.0.1:61215] 14:04:10.678 CachingConnectionFactory - Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)

Then several loops of this logging from consuming the message

[DEBUG] [events.service-1] 14:04:11.886 RabbitMqListener - Received event with id:foobar and deliveryTag:2
[DEBUG] [events.service-1] 14:04:11.886 RabbitMqListener - headers:{amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=, amqp_deliveryTag=2, amqp_consumerQueue=events.service, amqp_redelivered=false, amqp_receivedRoutingKey=events.service, x-exception-message=Fail to event with id:foobar, x-original-routingKey=events, amqp_timestamp=Wed Dec 14 14:03:58 GMT 2022, amqp_messageId=4ae864f8-0709-1fdc-b26d-611573474d60, x-original-exchange=events, id=9fe66161-d224-2988-7ddd-337975fec66a, amqp_consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, sourceData=(Body:'{"id":"foobar","timestamp":1671026638483}' MessageProperties [headers={x-exception-message=Fail to evaluate event with id:foobar, x-original-routingKey=events, x-original-exchange=events, x-exception-stacktrace=org.springframework.messaging.MessagingException: Exception thrown while invoking RabbitMqListener#process[4 args]; nested exception is org.springframework.amqp.AmqpRejectAndDontRequeueException: Fail to evaluate event with id:foobar, failedMessage=GenericMessage [payload=byte[359], headers={amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=events, amqp_deliveryTag=1, amqp_consumerQueue=events.service, amqp_redelivered=false, amqp_receivedRoutingKey=events, amqp_timestamp=Wed Dec 14 14:03:58 GMT 2022, amqp_messageId=4ae864f8-0709-1fdc-b26d-611573474d60, id=ca45340c-a895-61c7-2bc5-99b88b6b7e23, amqp_consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, sourceData=(Body:'{"id":"foobar","timestamp":1671026638483}' MessageProperties [headers={}, timestamp=Wed Dec 14 14:03:58 GMT 2022, messageId=4ae864f8-0709-1fdc-b26d-611573474d60, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=events, receivedRoutingKey=events, deliveryTag=1, consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, consumerQueue=events.service]), contentType=application/json, timestamp=1671026651701}]
[DEBUG] [events.service-1] 14:04:11.887 RabbitMqListener - x-death header:null
[ERROR] [events.service-1] 14:04:11.889 LoggingHandler - org.springframework.messaging.MessagingException: Exception thrown while invoking RabbitMqListener#process[4 args]; nested exception is org.springframework.amqp.AmqpRejectAndDontRequeueException: Fail to evaluate event with id:foobar, failedMessage=GenericMessage [payload=byte[359], headers={amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=, amqp_deliveryTag=2, amqp_consumerQueue=events.service, amqp_redelivered=false, amqp_receivedRoutingKey=events.service, x-exception-message=Fail to evaluate event with id:foobar, x-original-routingKey=events, amqp_timestamp=Wed Dec 14 14:03:58 GMT 2022, amqp_messageId=4ae864f8-0709-1fdc-b26d-611573474d60, x-original-exchange=events, id=9fe66161-d224-2988-7ddd-337975fec66a, amqp_consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, sourceData=(Body:'{"id":"foobar","timestamp":1671026638483}' MessageProperties [headers={x-exception-message=Fail to evaluate event with id:foobar, x-original-routingKey=events, x-original-exchange=events, x-exception-stacktrace=org.springframework.messaging.MessagingException: Exception thrown while invoking RabbitMqListener#process[4 args]; nested exception is org.springframework.amqp.AmqpRejectAndDontRequeueException: Fail to evaluate event with id:foobar, failedMessage=GenericMessage [payload=byte[359], headers={amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=events, amqp_deliveryTag=1, amqp_consumerQueue=events.service, amqp_redelivered=false, amqp_receivedRoutingKey=events, amqp_timestamp=Wed Dec 14 14:03:58 GMT 2022, amqp_messageId=4ae864f8-0709-1fdc-b26d-611573474d60, id=ca45340c-a895-61c7-2bc5-99b88b6b7e23, amqp_consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, sourceData=(Body:'{"id":"foobar","timestamp":1671026638483}' MessageProperties [headers={}, timestamp=Wed Dec 14 14:03:58 GMT 2022, messageId=4ae864f8-0709-1fdc-b26d-611573474d60, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=events, receivedRoutingKey=events, deliveryTag=1, consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, consumerQueue=events.service]), contentType=application/json, timestamp=1671026651701}]
    at org.springframework.cloud.stream.binding.StreamListenerMessageHandler.handleRequestMessage(StreamListenerMessageHandler.java:64)
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:520)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
    at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:208)
    at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$800(AmqpInboundChannelAdapter.java:65)
    at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.createAndSend(AmqpInboundChannelAdapter.java:322)
    at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.onMessage(AmqpInboundChannelAdapter.java:285)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1630)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1549)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1537)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1528)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1472)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:977)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:923)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:83)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1298)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1204)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Fail to evaluate event with id:foobar
    at com.foo.bar.listener.RabbitMqListener.processEvent(RabbitMqListener.java:99)
    at com.foo.bar.listener.RabbitMqListener.process(RabbitMqListener.java:70)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:171)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120)
    at org.springframework.cloud.stream.binding.StreamListenerMessageHandler.handleRequestMessage(StreamListenerMessageHandler.java:55)
    ... 27 more
Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: [no body]
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:113)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:169)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122)
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:639)
    at com.foo.bar.client.ClientImpl.readObject(ClientImpl.java:46)
    at com.foo.bar.client.ClientImpl$$FastClassBySpringCGLIB$$e4db9dde.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
    at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:53)
    at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:366)
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:421)
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:346)
    at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
    ... 35 more
}, timestamp=Wed Dec 14 14:03:58 GMT 2022, messageId=4ae864f8-0709-1fdc-b26d-611573474d60, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=, receivedRoutingKey=events.service, deliveryTag=3, consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, consumerQueue=events.service]), contentType=application/json, x-exception-stacktrace=org.springframework.messaging.MessagingException: Exception thrown while invoking RabbitMqListener#process[4 args]; nested exception is org.springframework.amqp.AmqpRejectAndDontRequeueException: Fail to evaluate event with id:foobar, failedMessage=GenericMessage [payload=byte[359], headers={amqp_receivedDeliveryMode=PERSISTENT, amqp_receivedExchange=events, amqp_deliveryTag=1, amqp_consumerQueue=events.service, amqp_redelivered=false, amqp_receivedRoutingKey=events, amqp_timestamp=Wed Dec 14 14:03:58 GMT 2022, amqp_messageId=4ae864f8-0709-1fdc-b26d-611573474d60, id=ca45340c-a895-61c7-2bc5-99b88b6b7e23, amqp_consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, sourceData=(Body:'{"id":"foobar","timestamp":1671026638483}' MessageProperties [headers={}, timestamp=Wed Dec 14 14:03:58 GMT 2022, messageId=4ae864f8-0709-1fdc-b26d-611573474d60, contentType=application/json, contentLength=0, receivedDeliveryMode=PERSISTENT, priority=0, redelivered=false, receivedExchange=events, receivedRoutingKey=events, deliveryTag=1, consumerTag=amq.ctag-QQ85gnjxKZvE5RTYUWJ1hg, consumerQueue=events.service]), contentType=application/json, timestamp=1671026651701}]
    at org.springframework.cloud.stream.binding.StreamListenerMessageHandler.handleRequestMessage(StreamListenerMessageHandler.java:64)
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:134)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:62)
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:570)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:520)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
    at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:208)
    at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$800(AmqpInboundChannelAdapter.java:65)
    at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.createAndSend(AmqpInboundChannelAdapter.java:322)
    at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.onMessage(AmqpInboundChannelAdapter.java:285)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1630)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1549)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1537)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1528)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1472)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:977)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:923)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:83)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1298)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1204)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: Failed to evaluate event with id:foobar
    at com.foo.bar.listener.RabbitMqListener.processEvent(RabbitMqListener.java:99)
    at com.foo.bar.listener.RabbitMqListener.process(RabbitMqListener.java:70)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:171)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120)
    at org.springframework.cloud.stream.binding.StreamListenerMessageHandler.handleRequestMessage(StreamListenerMessageHandler.java:55)
    ... 27 more
Caused by: org.springframework.web.client.HttpClientErrorException$NotFound: 404 Not Found: [no body]
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:113)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:169)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122)
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:639)
    at com.foo.bar.client.ClientImpl.readObject(ClientImpl.java:46)
    at com.foo.bar.client.ClientImpl$$FastClassBySpringCGLIB$$e4db9dde.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
    at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:53)
    at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:366)
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:421)
    at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:346)
    at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
    ... 35 more

We get several repeats of this loop before

[ERROR] [events.service-1] 14:04:11.951 SimpleMessageListenerContainer - Consumer thread error, thread abort.
[ERROR] [events.service-1] 14:04:11.954 SimpleMessageListenerContainer - Stopping container from aborted consumer
[INFO] [events.service-1] 14:04:11.954 SimpleMessageListenerContainer - Waiting for workers to finish.
[INFO] [events.service-1] 14:04:11.954 SimpleMessageListenerContainer - Successfully waited for workers to finish.


Solution

  • The x-death header is set by the broker, when it (the broker) routes a rejected message to the DLQ; by default, instead of rejecting a message, the binder republishes to the DLQ with additional information such as exception (which is not available when the broker does the routing).

    To use broker routing instead, set republishToDlq to false.