I'm trying to configure ActiveMQ Artemis HA with ZooKeeper lock manager using 2 machines. ZooKeeper runs on 3 machines. Here is the parts of broker.xml file:
<connectors>
<connector name="invm-connector">vm://0</connector>
<connector name="mq1">tcp://host-1:61616</connector>
<connector name="mq2">tcp://host-2:61616</connector>
</connectors>
<acceptors>
<acceptor name="mq1">tcp://host-1:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,OPENWIRE;supportAdvisory=false;useEpoll=true</acceptor>
<acceptor name="invm-acceptor">vm://0</acceptor>
</acceptors>
<cluster-connections>
<cluster-connection name="mycluster">
<connector-ref>mq1</connector-ref>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>mq2</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<colocated>
<max-backups>1</max-backups>
<request-backup>true</request-backup>
<backup-request-retries>-1</backup-request-retries>
<backup-request-retry-interval>5000</backup-request-retry-interval>
<excludes>
<connector-ref>mq1</connector-ref>
<connector-ref>mq2</connector-ref>
</excludes>
<primary>
<manager>
<properties>
<property key="connect-string" value="zoo-1:2181,zoo-2:2181,zoo-3:2181"/>
</properties>
</manager>
</primary>
<backup>
<manager>
<properties>
<property key="connect-string" value="zoo-1:2181,zoo-2:2181,zoo-3:2181"/>
</properties>
</manager>
</backup>
</colocated>
</replication>
</ha-policy>
The second broker.xml is the same (acceptors and cluster-connections just symmetric).
It's doubtful that in the broker's logs I don't see anything about connection to ZooKeeper or something related to it. There aren't even debug entries.
What could be a problem and why the broker does not connect to the ZooKeeper?
Here are the logs about one of the running broker:
2024-05-17 14:11:49,902 WARN [org.apache.activemq.artemis.core.server] AMQ224110: Configuration 'whitelist' is deprecated, please use the 'allowlist' configuration
_ _ _
/ \ ____| |_ ___ __ __(_) _____
/ _ \| _ \ __|/ _ \ \/ | |/ __/
/ ___ \ | \/ |_/ __/ |\/| | |\___ \
/_/ \_\| \__\____|_| |_|_|/___ /
Apache ActiveMQ Artemis 2.33.0
2024-05-17 14:11:50,952 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server version 2.33.0
2024-05-17 14:11:51,004 INFO [org.apache.activemq.artemis.core.server] AMQ221000: Primary message broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2024-05-17 14:11:51,070 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2024-05-17 14:11:51,296 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1073741824
2024-05-17 14:11:51,378 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2024-05-17 14:11:51,379 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2024-05-17 14:11:51,380 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2024-05-17 14:11:51,380 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2024-05-17 14:11:51,381 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2024-05-17 14:11:51,381 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2024-05-17 14:11:52,752 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address DLQ supporting [ANYCAST]
2024-05-17 14:11:52,754 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue DLQ on address DLQ
2024-05-17 14:11:52,759 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address ExpiryQueue supporting [ANYCAST]
2024-05-17 14:11:52,759 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue ExpiryQueue on address ExpiryQueue
2024-05-17 14:11:52,760 INFO [org.apache.activemq.artemis.core.server] AMQ221080: Deploying address TEST supporting [ANYCAST]
2024-05-17 14:11:52,760 INFO [org.apache.activemq.artemis.core.server] AMQ221003: Deploying ANYCAST queue TEST on address TEST
2024-05-17 14:11:53,225 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:11:53,225 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:11:53,284 INFO [org.apache.activemq.artemis.core.server] AMQ221020: Started EPOLL Acceptor at 10.103.41.254:61616 for protocols [CORE,OPENWIRE]
2024-05-17 14:11:53,284 INFO [org.apache.activemq.artemis.core.server] AMQ221007: Server is now active
2024-05-17 14:11:53,284 INFO [org.apache.activemq.artemis.core.server] AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.33.0 [mq1, nodeID=833a7f68-f0e8-11eb-beec-fa163e1c565e]
2024-05-17 14:11:53,293 INFO [org.apache.activemq.artemis] AMQ241003: Starting embedded web server
2024-05-17 14:11:53,953 INFO [org.apache.activemq.hawtio.branding.PluginContextListener] Initialized activemq-branding plugin
2024-05-17 14:11:54,049 INFO [org.apache.activemq.hawtio.plugin.PluginContextListener] Initialized artemis-plugin plugin
2024-05-17 14:11:54,598 INFO [io.hawt.HawtioContextListener] Initialising hawtio services
2024-05-17 14:11:54,626 INFO [io.hawt.system.ConfigManager] Configuration will be discovered via system properties
2024-05-17 14:11:54,627 INFO [io.hawt.jmx.JmxTreeWatcher] Welcome to Hawtio 2.17.7
2024-05-17 14:11:54,645 INFO [io.hawt.web.auth.AuthenticationConfiguration] Starting hawtio authentication filter, JAAS realm: "activemq" authorized role(s): "amq" role principal classes: "org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal"
2024-05-17 14:11:54,679 INFO [io.hawt.web.auth.LoginRedirectFilter] Hawtio loginRedirectFilter is using 1800 sec. HttpSession timeout
2024-05-17 14:11:54,709 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server started at http://0.0.0.0:8161
2024-05-17 14:11:54,711 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia REST API available at http://0.0.0.0:8161/console/jolokia
2024-05-17 14:11:54,711 INFO [org.apache.activemq.artemis] AMQ241004: Artemis Console available at http://0.0.0.0:8161/console
2024-05-17 14:11:58,226 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:11:58,227 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:03,228 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:03,228 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:08,229 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:08,229 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:13,229 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:13,230 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:18,230 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:18,231 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:23,232 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:23,232 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:28,233 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:28,233 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:33,233 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:33,234 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:38,234 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:38,235 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:43,235 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:43,235 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:48,236 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:48,236 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:53,236 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:53,237 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:12:58,237 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:12:58,237 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:13:03,238 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:13:03,238 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:13:08,239 INFO [org.apache.activemq.artemis.core.server] AMQ221066: Initiating quorum vote: RequestBackupQuorumVote
2024-05-17 14:13:08,239 INFO [org.apache.activemq.artemis.core.server] AMQ221084: Requested 0 quorum votes
2024-05-17 14:13:08,992 INFO [org.apache.activemq.artemis.core.server] AMQ221062: Received quorum vote request: RequestBackupVote [backupsSize=-1, nodeID=null, backupAvailable=false]
2024-05-17 14:13:08,992 INFO [org.apache.activemq.artemis.core.server] AMQ221063: Sending quorum vote response: RequestBackupVote [backupsSize=0, nodeID=833a7f68-f0e8-11eb-beec-fa163e1c565e, backupAvailable=true]
2024-05-17 14:13:09,272 INFO [org.apache.activemq.artemis.core.server] AMQ221000: Backup message broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journalcolocated_backup_01,bindingsDirectory=data/bindingscolocated_backup_01,largeMessagesDirectory=data/large-messagescolocated_backup_01,pagingDirectory=data/pagingcolocated_backup_01)
2024-05-17 14:13:09,282 INFO [org.apache.activemq.artemis.core.server] AMQ221049: Activating Replica for node: fee8a057-f0e4-11eb-8d04-fa163e943492
2024-05-17 14:13:09,286 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/activemq/apache-artemis-2.17.0/bin/mybroker/data/bindingscolocated_backup_01/oldreplica.7
2024-05-17 14:13:09,288 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/activemq/apache-artemis-2.17.0/bin/mybroker/data/bindingscolocated_backup_01 to /opt/activemq/apache-artemis-2.17.0/bin/mybroker/data/bindingscolocated_backup_01/oldreplica.9
2024-05-17 14:13:09,288 INFO [org.apache.activemq.artemis.core.server] AMQ221055: There were too many old replicated folders upon startup, removing /opt/activemq/apache-artemis-2.17.0/bin/mybroker/data/journalcolocated_backup_01/oldreplica.25
2024-05-17 14:13:09,289 INFO [org.apache.activemq.artemis.core.server] AMQ222162: Moving data directory /opt/activemq/apache-artemis-2.17.0/bin/mybroker/data/journalcolocated_backup_01 to /opt/activemq/apache-artemis-2.17.0/bin/mybroker/data/journalcolocated_backup_01/oldreplica.27
2024-05-17 14:13:09,290 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2024-05-17 14:13:09,304 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
2024-05-17 14:13:09,306 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
2024-05-17 14:13:09,307 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
2024-05-17 14:13:09,308 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT
2024-05-17 14:13:09,312 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE
2024-05-17 14:13:09,314 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
2024-05-17 14:13:09,329 INFO [org.apache.activemq.artemis.core.server] AMQ221109: Apache ActiveMQ Artemis Backup Server version 2.33.0 [null] started; waiting for primary to fail before activating
Also, I would be very thankful if you'll give any advice about ways to debug voting process with ZooKeeper to understand it clearly.
I did try to start up an ActiveMQ Artemis server and expected to see any logs about connecting to ZooKeeper and didn't see anything about it.
As it was recommended by Justin I've reconfigured cluster for primary:
<ha-policy>
<replication>
<primary>
<group-name>group1</group-name>
<manager>
<properties>
<property key="connect-string" value="zoo1:2181,zoo2:2181,zoo3:2181"/>
</properties>
</manager>
</primary>
</replication>
</ha-policy>
and for backup:
<ha-policy>
<replication>
<backup>
<group-name>group1</group-name>
<manager>
<properties>
<property key="connect-string" value="zoo1:2181,zoo2:2181,zoo3:2181"/>
</properties>
</manager>
<allow-failback>true</allow-failback>
</backup>
</replication>
</ha-policy>
And it works fine now. Thank you!