During high load MQQueueConnectionFactory suddenly change port to default port 1414

Question

During high load, we see that the MQQueueConnectionFactory suddenly trying to connect to default MQ port 1414 instead of the assigned port.

Background:
We use Websphere 9.0.5.13 to deploy an ear file. Java version: 8.0.7.15. The server in this Websphere communicate to external IBM MQ. We use IBM MQ library version: com.ibm.mq.allclient-9.2.5.0.jar. We connect to 3 different IBM MQ queue managers and hundreds of channels. We don't use IBM MQ default port numbers i.e. 1414. Instead, we override it to some other ports. We always use new MQQueueConnectionFactory before we send to a queue, and close the sender, session, and connection afterwards.

Issue:
At the start of server, we process multiple queues simultaneously. Most of the time, this causes:

JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2538' ('MQRC_HOST_NOT_AVAILABLE').

Digging deeper, it is because the MQQueueConnectionFactory try to connect to default port 1414 instead of the correct port we have configured.

Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2538;AMQ9204: Connection to host 'ecpmq2(1414)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2538;AMQ9204: Connection to host 'hostname******/ipaddr***...:1414' rejected. [1=java.net.ConnectException[A remote host did not respond within the timeout period. (Connection timed out)],3=hostname***/ipaddr***...:1414,4=TCP,5=Socket.connect]],3=hostname***(1414),4=,5=RemoteTCPConnection.bindAndConnectSocket]

I tried to replicate the scenario in local PC and our test env, but to no avail.

Sample Code:
This is the code that we have. Yes, the code is farcry from being efficient, but it still does not explain why suddenly the MQ try to connect to default port.

        for (String msg : messages) {
            MQQueueConnectionFactory queueConnectionFactory = null;
            MQQueueConnection mqConn = null;
            try {
                queueConnectionFactory = new MQQueueConnectionFactory();
                queueConnectionFactory.setQueueManager(queueInfo.getQueueManager());
                queueConnectionFactory.setHostName(queueInfo.getHostname());
                queueConnectionFactory.setPort(queueInfo.getPort());
                queueConnectionFactory.setChannel(queueInfo.getChannel());
                queueConnectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
                
                mqConn = (MQQueueConnection) queueConnectionFactory.createQueueConnection();
                mqConn.start();
                sendSingleMsg(mqConn, queueInfo.getQueueName(), msg);
            } catch (Exception e) {
                log.error(e.getMessage(), e);
                printQueueDetails(queueConnectionFactory);
            } finally {
                if (mqConn != null) {
                    try {
                        mqConn.close();
                    } catch (JMSException ex) {
                        log.error(ex.getMessage(), ex);
                    }
                    mqConn = null;
                }
            }
        }

Question:
Is it possible that MQQueueConnectionFactory fail somewhere due to high concurrency and set the port to default port instead of the assigned port? Does it keep some failover set of hostnames and it includes the default port?

Answer 1

An update on this item. I put logging in various place to log the port number. Then we found out that the port in queueConnectionFactory was intermittently reset to default port after calling queueConnectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);

After updating the log level of ibm mq to TRACE, I found that new MQ library com.ibm.mq.allclient-9.2.5.0.jar has a portSet flag that would be set via a thread in somewhere in setPort(). Since it is set using a thread, and not via a sequential method call, there is a chance the thread does not start or start later than the current thread that will call queueConnectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);. If that happens, the port would be reset to default value. This is the snippet of what happen deep inside setPort()

if (parent instanceof MQConnectionFactory) {
                AccessController.doPrivileged(new PrivilegedAction<Object>() {
                    public Object run() {
                        if (Trace.isOn) {
                            Trace.entry(this, "com.ibm.msg.client.wmq.factories.WMQPortPropertyValidator", "run()");
                        }

                        try {
                            Field portSetField = MQConnectionFactory.class.getDeclaredField("portSet");
                            portSetField.setAccessible(true);
                            portSetField.setBoolean(parent, true);

and this is the snippet of what happen inside queueConnectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);

if (!this.portSet) {
  this.setPort(1414);
  this.portSet = false;
}

The fix is VERY SIMPLE! Just need to call setTransportType before calling setPort

queueConnectionFactory = new MQQueueConnectionFactory();
// call setTransportType first
queueConnectionFactory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
queueConnectionFactory.setQueueManager(queueInfo.getQueueManager());
queueConnectionFactory.setHostName(queueInfo.getHostname());
queueConnectionFactory.setChannel(queueInfo.getChannel());
// call setPort last
queueConnectionFactory.setPort(queueInfo.getPort());
mqConn = (MQQueueConnection) queueConnectionFactory.createQueueConnection();