Search code examples
javajmsactivemq-classicfailovercluster

ActiveMQ: How to handle broker failovers while using temporary queues

On my JMS applications we use temporary queues on Producers to be able to receive replies back from Consumer applications.

I am facing exactly same issue on my end as mentioned in this thread: http://activemq.2283324.n4.nabble.com/jira-Created-AMQ-3336-Temporary-Destination-errors-on-H-A-failover-in-broker-network-with-Failover-tt-td3551034.html#a3612738

Whenever I restarted an arbitrary broker in my network, I was getting many errors like this in my Consumer application log while trying to send reply to a temporary queue:

javax.jms.InvalidDestinationException:
  Cannot publish to a deleted Destination: temp-queue://ID:...

Then I saw Gary's response there suggesting to use

jms.watchTopicAdvisories=false

as a url param on the client brokerURL. I promptly changed my client broker URLs with this additional parameter. However now I am seeing errors like this when I restart my brokers in network for this failover testing:

javax.jms.JMSException: 
  The destination temp-queue:
    //ID:client.host-65070-1308610734958-2:1:1 does not exist.

I am using ActiveMQ 5.5 version. And my client broker URL looks like this:

failover:(tcp://amq-host1:61616,tcp://amq-host2.tred.aol.com:61616,tcp://amq-host3:61616,tcp://amq-host4:61616)?jms.useAsyncSend=true&timeout=5000&jms.watchTopicAdvisories=false

Additionally here is my activemq config XML for one of the 4 brokers: amq1.xml

Can someone here please look into this problem and suggest me what mistake I am making in this setup.

Update:

To clarify further on how I am doing request-response in my code:

  1. I already use a per producer destination (i.e. temporary queue) and set this in reply-to header of every message.
  2. I am already sending a per message unique correlation identifier in JMSCorrelationID header.
  3. As far as I know even Camel and Spring are also using temporary queue for request-response mechanism. Only difference is that Spring JMS implementation creates and destroys temporary queue for every message whereas I create temporary queue for the lifetime of the producer. This temporary queue is destroyed when client (producer) app shutsdown or by the AMQ broker when it realizes there are no active producer attached with this temporary queue.
  4. I am already setting a message expiry on each message on Producer side so that message is not held up in a queue for too long (60 sec).
Solution

  • There is a broker attribute, org.apache.activemq.broker.BrokerService#cacheTempDestinations that should help in the failover: case. Set that to true in xml configuration, and a temp destination will not be removed immediately when a client disconnects. A fast failover: reconnect will be able to producer and/or consume from the temp queue again.

    There is a timer task based on timeBeforePurgeTempDestinations (default 5 seconds) that handles cache removal.

    One caveat though, I don't see any tests in activemq-core that make use of that attribute so I can't give you any guarantee on this one.