Message driven bean does not receive message from other nodes in the cluster

I have a Wildfly 23 cluster with two nodes (node-1, node-2) running the standalone-full-ha profile. The two cluster nodes boot and communicate with each other correctly (as far as I can judge).

My intention is to send a JMS message on a topic from node-1 and have message driven beans (MDB) on node-1 and node-2 consume this message.

The MDB code:

import javax.ejb.ActivationConfigProperty;
import javax.ejb.MessageDriven;
import javax.jms.Message;
import javax.jms.MessageListener;

@MessageDriven(activationConfig = {
        @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"), 
        @ActivationConfigProperty(propertyName = "destinationLookup", 
            propertyValue = "java:/jms/topic/myTopic"), 
        @ActivationConfigProperty(propertyName = "maxSession", propertyValue = "1")
})
public class ClusteredEventListener implements MessageListener {

    @Override
    public void onMessage(final Message message) {
        // consume message
    }

}

The message publisher code:

import java.io.Serializable;

import javax.annotation.Resource;
import javax.ejb.Startup;
import javax.enterprise.context.ApplicationScoped;
import javax.jms.JMSException;
import javax.jms.MessageProducer;
import javax.jms.ObjectMessage;
import javax.jms.Session;
import javax.jms.Topic;
import javax.jms.TopicConnection;
import javax.jms.TopicConnectionFactory;
import javax.jms.TopicSession;

@Startup
@ApplicationScoped
public class ClusteredEventSender {

    @Resource(lookup = "java:/jms/topic/myTopic")
    private Topic topic;

    @Resource(mappedName = "java:/ConnectionFactory")
    private TopicConnectionFactory connectionFactory;

    public void broadcast(final Serializable event) {
        try {
            try (TopicConnection connection = this.connectionFactory.createTopicConnection()) { 
                try (TopicSession session = connection.createTopicSession(false, Session.AUTO_ACKNOWLEDGE)) {
                    try (MessageProducer messageProducer = session.createPublisher(this.topic)) { 
                        final ObjectMessage message = session.createObjectMessage(event);
                        messageProducer.send(message);
                    }
                }
            }
        } catch (final JMSException e) {
            log.error("Could not broadcast event to topic: " + event, e);
        }
    }

}

Snippet from standalone.xml:

<subsystem xmlns="urn:jboss:domain:messaging-activemq:13.0">
    <server name="default">
        ...
        <jms-topic name="myTopic" entries="java:/jms/topic/myTopic"/>
        ...
    </server>
</subsystem>

Outcome is that the messages are only consumed on the node that generated them. The other node does not receive any message.

Experiments

I experimented by using java:jboss/exported/jms/RemoteConnectionFactory with durable subscripitions, unique clientID and subscriptionName for each node and a user "jmsuser", and using a topic in java:jboss/exported (java:jboss/exported/jms/topic/myTopic), changing/extending the annotations on the MDB as follows:

...
@ActivationConfigProperty(propertyName = "destinationLookup", propertyValue = "java:jboss/exported/jms/topic/myTopic"),
@ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue = "Durable"), 
@ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "subscription-${jboss.node.name}"), 
@ActivationConfigProperty(propertyName = "clientID", propertyValue = "node-${jboss.node.name}"), 
@ActivationConfigProperty(propertyName = "connectionFactoryLookup", propertyValue = "java:jboss/exported/jms/RemoteConnectionFactory"), 
@ActivationConfigProperty(propertyName = "user", propertyValue = "jmsuser"), 
@ActivationConfigProperty(propertyName = "password", propertyValue = "jmsuser")
...

Note: The "jmsuser" was added using the batch script add-user.bat in the wildfly/bin directory. It has been assigned the role "guest". The guest role has been extended regarding durable queues. The property replacement in annotations (to make the ${jboss.node.name} in clientID and subscriptionName work) has been activated in standalone.xml:

<subsystem xmlns="urn:jboss:domain:ee:6.0">
    ...
    <annotation-property-replacement>true</annotation-property-replacement>
    ...
</subsystem>
...
<subsystem xmlns="urn:jboss:domain:messaging-activemq:13.0">
    <server name="default">
        ...
        <security-setting name="#">
            ...
            <role name="guest" delete-non-durable-queue="true"
               create-non-durable-queue="true"
               delete-durable-queue="true"
               create-durable-queue="true"
               consume="true"
               send="true" />
            ...
        </security-setting>
        ...
        <jms-topic name="myTopic" entries="java:jboss/exported/jms/topic/myTopic"/>
        ...
    </server>
</subsystem>
...

The amended message publisher code:

public class ClusteredEventSender {

    @Resource(lookup = "java:jboss/exported/jms/topic/myTopic")
    private Topic topic;

    @Resource(lookup = "java:jboss/exported/jms/RemoteConnectionFactory")
    private TopicConnectionFactory connectionFactory;

    public void broadcast(final Serializable event) {
        try {
            try (TopicConnection connection = this.connectionFactory.createTopicConnection("jmsuser", "jmsuser")) {
                try (TopicSession session = connection.createTopicSession(false, Session.AUTO_ACKNOWLEDGE)) {
                    try (MessageProducer messageProducer = session.createPublisher(this.topic)) { 
                        final ObjectMessage message = session.createObjectMessage(event);
                        messageProducer.send(message);
                    }
                }
            }
        } catch (final JMSException e) {
            log.error("Could not broadcast event to topic: " + event, e);
        }
    }

}

Experiments outcome: The connection to the RemoteConnectionFactory works, but nevertheless the behaviour remains the same as before.

My questions are: How can a publish / subscribe using JMS / ActiveMQ be implemented in a Wildfly cluster? What does the message driven bean look like, and how are the messages to be sent? What configuration is needed?

Solution

Thanks to @ehsavoie's hint we managed to solve the problem. Solution in short:

Using java:jboss/exported/jms/topic/myTopic or java:jboss/exported/jms/RemoteConnectionFactory is not necessary. Neither are any @ActivationConfigProperty-Entries for subscriptionName, clientId, user or password. All experiments described in the question were a dead end.
Creating a connector from node 1 to node 2 (as suggested in the comment) had some effect, since then node 1 was able to send events to node 2. Nevertheless, since we were using JGroups discovery (as in full-ha) this was merely a clue that the discovery was not working correctly.
After changing the cluster password from the default "CHANGE ME!!" (duh!) and adding the attribute jms-connection-factory="java:jboss/DefaultJMSConnectionFactory" to the <default-bindings> in our standalone.xml everything worked as expected. Those two little rascals slipped away during setup.

Thus, the relevant standalone.xml snippets are:

<cluster password="something else than CHANGE ME!!"/>

and

<default-bindings ... jms-connection-factory="java:jboss/DefaultJMSConnectionFactory" ... />

Update

For better orientation: The locations of the above mentioned snippets are:

<server xmlns="urn:jboss:domain:16.0">
    ...
    <profile>
        ...
        <subsystem xmlns="urn:jboss:domain:messaging-activemq:13.0">
            <server name="default">
                <cluster .../>
                ...
            </server>
        </subsystem>
        ...
        <subsystem xmlns="urn:jboss:domain:ee:6.0">
            <default-bindings .../>
        </subsystem>
        ...
    </profile>
    ...
</server>