I'm creating ActiveMQ Artemis cluster using JGroup's Kubernetes Ping. After I have upgraded to 2.33.0 I'm getting java.lang.NoSuchMethodError. Seems like JGroups to 5.3.2.Final has to do something.
2024-04-28 06:51:47,578 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server version 2.33.0
2024-04-28 06:51:47,603 INFO [org.apache.activemq.artemis.core.server] AMQ221000: Primary message broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=data/journal,bindingsDirectory=data/bindings,largeMessagesDirectory=data/large-messages,pagingDirectory=data/paging)
2024-04-28 06:51:47,721 WARN [org.jgroups.stack.Configurator] CENTRAL_LOCK has been deprecated; please upgrade to a newer version of the protocol
2024-04-28 06:51:47,774 INFO [org.jgroups.protocols.kubernetes.KUBE_PING] namespace amq-ap-dev set; clustering enabled
2024-04-28 06:51:47,779 INFO [org.jgroups.JChannel] local_addr: e0449f9d-ef60-4a9d-b412-da5b85719e3d, name: activemq-1-statefulset-0-57906
-------------------------------------------------------------------
GMS: address=activemq-1-statefulset-0-57906, cluster=active_broadcast_channel, physical address=10.1.0.159:7800
-------------------------------------------------------------------
2024-04-28 06:51:51,061 INFO [org.jgroups.protocols.pbcast.GMS] activemq-1-statefulset-0-57906: no members discovered after 3273 ms: creating cluster as coordinator
2024-04-28 06:52:01,219 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal
2024-04-28 06:52:01,270 WARN [org.jgroups.stack.Configurator] CENTRAL_LOCK has been deprecated; please upgrade to a newer version of the protocol
2024-04-28 06:52:01,275 INFO [org.jgroups.protocols.kubernetes.KUBE_PING] namespace amq-ap-dev set; clustering enabled
2024-04-28 06:52:01,282 INFO [org.jgroups.JChannel] local_addr: 6ab26800-294a-4629-9250-40a98a4a2e92, name: activemq-1-statefulset-0-58068
-------------------------------------------------------------------
GMS: address=activemq-1-statefulset-0-58068, cluster=active_broadcast_channel, physical address=10.1.0.159:7800
-------------------------------------------------------------------
2024-04-28 06:52:01,349 ERROR [org.apache.activemq.artemis.core.server] AMQ224097: Failed to start server
java.lang.NoSuchMethodError: 'org.jgroups.util.ByteArray org.jgroups.protocols.kubernetes.KUBE_PING.marshal(org.jgroups.protocols.PingData)'
at org.jgroups.protocols.kubernetes.KUBE_PING.findMembers(KUBE_PING.java:278) ~[jgroups-kubernetes-2.0.2.Final.jar:?]
at org.jgroups.protocols.Discovery.callFindMembersInAllDiscoveryProtocols(Discovery.java:393) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.Discovery.findMembers(Discovery.java:240) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.Discovery.down(Discovery.java:430) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.MERGE3.down(MERGE3.java:268) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:411) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.FailureDetection.down(FailureDetection.java:159) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:102) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.BARRIER.down(BARRIER.java:138) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:635) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:653) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:275) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.pbcast.ClientGmsImpl.joinInternal(ClientGmsImpl.java:67) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:37) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:900) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.FlowControl.down(FlowControl.java:201) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.stack.Protocol.down(Protocol.java:309) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.FRAG2.down(FRAG2.java:103) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:205) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.COUNTER.down(COUNTER.java:241) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.protocols.Locking.down(Locking.java:226) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:947) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.JChannel.down(JChannel.java:601) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.JChannel._connect(JChannel.java:803) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.JChannel.connect(JChannel.java:326) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.jgroups.JChannel.connect(JChannel.java:317) ~[jgroups-5.3.2.Final.jar:5.3.2.Final]
at org.apache.activemq.artemis.api.core.jgroups.JChannelWrapper.connect(JChannelWrapper.java:127) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.api.core.JGroupsBroadcastEndpoint.internalOpen(JGroupsBroadcastEndpoint.java:124) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.api.core.JGroupsBroadcastEndpoint.openClient(JGroupsBroadcastEndpoint.java:102) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.cluster.DiscoveryGroup.start(DiscoveryGroup.java:113) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.startDiscovery(ServerLocatorImpl.java:298) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.initialize(ServerLocatorImpl.java:282) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterController.configAndAdd(ClusterController.java:245) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterController.addClusterConnection(ClusterController.java:207) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterManager.deployClusterConnection(ClusterManager.java:632) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterManager.deploy(ClusterManager.java:249) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:3324) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryActivation.run(SharedNothingPrimaryActivation.java:115) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:738) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:628) [artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66) [artemis-cli-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) [artemis-cli-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:221) [artemis-cli-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167) [artemis-cli-2.33.0.jar:2.33.0]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:157) [artemis-boot.jar:2.33.0]
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) [artemis-boot.jar:2.33.0]
java.lang.NoSuchMethodError: 'org.jgroups.util.ByteArray org.jgroups.protocols.kubernetes.KUBE_PING.marshal(org.jgroups.protocols.PingData)'
at org.jgroups.protocols.kubernetes.KUBE_PING.findMembers(KUBE_PING.java:278)
at org.jgroups.protocols.Discovery.callFindMembersInAllDiscoveryProtocols(Discovery.java:393)
at org.jgroups.protocols.Discovery.findMembers(Discovery.java:240)
at org.jgroups.protocols.Discovery.down(Discovery.java:430)
at org.jgroups.protocols.MERGE3.down(MERGE3.java:268)
at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:411)
at org.jgroups.protocols.FailureDetection.down(FailureDetection.java:159)
at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:102)
at org.jgroups.protocols.BARRIER.down(BARRIER.java:138)
at org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:635)
at org.jgroups.protocols.UNICAST3.down(UNICAST3.java:653)
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:275)
at org.jgroups.protocols.pbcast.ClientGmsImpl.joinInternal(ClientGmsImpl.java:67)
at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:37)
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:900)
at org.jgroups.protocols.FlowControl.down(FlowControl.java:201)
at org.jgroups.stack.Protocol.down(Protocol.java:309)
at org.jgroups.protocols.FRAG2.down(FRAG2.java:103)
at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:205)
at org.jgroups.protocols.COUNTER.down(COUNTER.java:241)
at org.jgroups.protocols.Locking.down(Locking.java:226)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:947)
at org.jgroups.JChannel.down(JChannel.java:601)
at org.jgroups.JChannel._connect(JChannel.java:803)
at org.jgroups.JChannel.connect(JChannel.java:326)
at org.jgroups.JChannel.connect(JChannel.java:317)
at org.apache.activemq.artemis.api.core.jgroups.JChannelWrapper.connect(JChannelWrapper.java:127)
at org.apache.activemq.artemis.api.core.JGroupsBroadcastEndpoint.internalOpen(JGroupsBroadcastEndpoint.java:124)
at org.apache.activemq.artemis.api.core.JGroupsBroadcastEndpoint.openClient(JGroupsBroadcastEndpoint.java:102)
at org.apache.activemq.artemis.core.cluster.DiscoveryGroup.start(DiscoveryGroup.java:113)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.startDiscovery(ServerLocatorImpl.java:298)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.initialize(ServerLocatorImpl.java:282)
at org.apache.activemq.artemis.core.server.cluster.ClusterController.configAndAdd(ClusterController.java:245)
at org.apache.activemq.artemis.core.server.cluster.ClusterController.addClusterConnection(ClusterController.java:207)
at org.apache.activemq.artemis.core.server.cluster.ClusterManager.deployClusterConnection(ClusterManager.java:632)
at org.apache.activemq.artemis.core.server.cluster.ClusterManager.deploy(ClusterManager.java:249)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:3324)
at org.apache.activemq.artemis.core.server.impl.SharedNothingPrimaryActivation.run(SharedNothingPrimaryActivation.java:115)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:738)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:628)
at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66)
at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130)
at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:221)
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:157)
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64)
This is the JGroups XML I'm using:
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP
external_addr="${JGROUPS_EXTERNAL_ADDR:match-interface:eth0}"
bind_addr="site_local,match-interface:eth0"
bind_port="${TCP_PORT:7800}"
recv_buf_size="5M"
send_buf_size="1M"
thread_naming_pattern="cl"
thread_pool.min_threads="2"
thread_pool.max_threads="500"
thread_pool.keep_alive_time="30000" />
<RED/>
<org.jgroups.protocols.kubernetes.KUBE_PING port_range="0" namespace="${KUBERNETES_NAMESPACE:amq-ap-dev}"
labels="app=activemq-active-active" masterHost="kubernetes.default.svc" masterPort="443" masterProtocol="https"
caCertFile="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
saTokenFile="/var/run/secrets/kubernetes.io/serviceaccount/token"
/>
<MERGE3 max_interval="30000"
min_interval="10000"/>
<FD_SOCK external_addr="${JGROUPS_EXTERNAL_ADDR}"
start_port="${FD_SOCK_PORT:9000}"/>
<FD_ALL timeout="30000" interval="5000"/>
<VERIFY_SUSPECT timeout="1500" />
<BARRIER />
<pbcast.NAKACK2 xmit_interval="500"
xmit_table_num_rows="100"
xmit_table_msgs_per_row="2000"
xmit_table_max_compaction_time="30000"
use_mcast_xmit="false"
discard_delivered_msgs="true" />
<UNICAST3
xmit_table_num_rows="100"
xmit_table_msgs_per_row="1000"
xmit_table_max_compaction_time="30000"/>
<pbcast.STABLE desired_avg_gossip="50000"
max_bytes="8m"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000"/>
<MFC max_credits="2M"
min_threshold="0.4"/>
<FRAG2 frag_size="60K" />
<pbcast.STATE_TRANSFER />
<COUNTER/>
<CENTRAL_LOCK />
</config>
I have added these jars for Kubernetes JGroups in the lib folder of the ActiveMQ Artemis instance:
jgroups-kubernetes-2.0.2.Final.jar
kubernetes-client-6.12.1.jar
What jar should I add not to get this issue?
It appears you're hitting a bug in JGroups 5.3.2.Final. The method in question existed in 5.3.1.Final. Then it was removed in 5.3.2.Final and finally added back in 5.3.3.Final.
If you download JGroups 5.3.3.Final and deploy that to ActiveMQ Artemis then that should solve the issue.
This will be fixed in ActiveMQ Artemis 2.34.0 thanks to ARTEMIS-4728.