Search code examples
javaignite

Ignite Client Fails to Establish Connection to Cluster, Causes JVM to be Halted Immediately


I am working on a project that already was using ignite but with have run into a new issue.

The Ignite client successfully establishes connection to the cluster and begins initiating the caches. After cache initiation the ignite client crashes, shuts down the JVM and consequently kills the spawning process.

I am not aware of any changes that have occurred in my configuration and I have not upgraded the Ignite version (Ignite 2.12.0), any tips on what may be wrong would be apreciated.

Ignite Client Logs:

22-Aug-2022 21:39:48.082 INFO [disco-notifier-worker-#38%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.info Assigned mvcc coordinator [crd=MvccCoordinator [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nodeId=8280e085-0fdb-4288-9b4e-75779ac8e961, ver=1661204058977, local=false, initialized=false]]
22-Aug-2022 21:39:48.105 INFO [sys-#43%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.info Writing cluster ID and tag to metastorage on ready for write ClusterIdAndTag [id=c05dad9a-56f1-4e90-9a8e-2ef8f627c1b7, tag=angry_newton]
22-Aug-2022 21:39:48.106 WARNING [main] org.apache.ignite.logger.java.JavaLogger.warning Local node's value of 'java.net.preferIPv4Stack' system property differs from remote node's (all nodes in topology should have identical value) [locPreferIpV4=false, rmtPreferIpV4=true, locId8=535cb96f, rmtId8=8280e085, rmtAddrs=[**.**.**.**/**.**.**.**, /127.0.0.1], rmtNode=ClusterNode [id=8280e085-0fdb-4288-9b4e-75779ac8e961, order=1, addr=[**.**.**.**, 127.0.0.1], daemon=false]]
22-Aug-2022 21:39:48.109 WARNING [main] org.apache.ignite.logger.java.JavaLogger.warning Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=4801MB, available=8156MB]
22-Aug-2022 21:39:48.278 INFO [exchange-worker-#45%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.info Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], crd=false, evt=NODE_JOINED, evtNode=535cb96f-3e0b-402d-bdcc-128446028ca5, customEvt=null, allowMerge=true, exchangeFreeSwitch=false]
22-Aug-2022 21:39:48.279 INFO [exchange-worker-#45%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.info Components activation performed in 0 ms.
22-Aug-2022 21:39:48.610 INFO [sys-#50%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.info Started cache [name=FOOD_CACHE, id=251327108, dataRegionName=null, mode=REPLICATED, atomicity=ATOMIC, backups=2147483647, mvcc=false]
...
22-Aug-2022 21:39:48.620 INFO [sys-#47%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.info Started cache [name=BAR_CACHE, id=-2108476111, dataRegionName=null, mode=REPLICATED, atomicity=ATOMIC, backups=2147483647, mvcc=false]
22-Aug-2022 21:39:48.621 INFO [exchange-worker-#45%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.info Starting caches on local join performed in 343 ms.
22-Aug-2022 21:39:48.648 SEVERE [exchange-worker-#45%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.error Failed to send message to remote node [node=TcpDiscoveryNode [id=8280e085-0fdb-4288-9b4e-75779ac8e961, consistentId=**.**.**.**,127.0.0.1:47500, addrs=ArrayList [**.**.**.**, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, /**.**.**.**:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1661204387800, loc=false, ver=2.12.0#20220108-sha1:b1289f75, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null, partsSizes=null, partHistCntrs=null, err=null, client=true, exchangeStartTime=1661204388092, finishMsg=null, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=535cb96f-3e0b-402d-bdcc-128446028ca5, consistentId=535cb96f-3e0b-402d-bdcc-128446028ca5, addrs=ArrayList [0:0:0:0:0:0:0:1, **.**.**.**, 127.0.0.1], sockAddrs=HashSet [/[0:0:0:0:0:0:0:1]:0, /127.0.0.1:0, /**.**.**.**:0], discPort=0, order=4, intOrder=0, lastExchangeTime=1661204387636, loc=true, ver=2.12.0#20220108-sha1:b1289f75, isClient=true], topVer=4, msgTemplate=null, span=org.apache.ignite.internal.processors.tracing.NoopSpan@1d6ea3e0, nodeId8=535cb96f, msg=null, type=NODE_JOINED, tstamp=1661204388076], nodeId=535cb96f, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1661204387076, nodeOrder=0], super=GridCacheMessage [msgId=1, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], err=null, skipPrepare=false]]]]]
        class org.apache.ignite.IgniteCheckedException: No session found
                at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:660)
                at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:693)
                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1264)
                at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:691)
                at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:481)
                at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:240)
                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1195)
                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1142)
                at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2100)
                at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2193)
                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1266)
                at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:2201)
                at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.clientOnlyExchange(GridDhtPartitionsExchangeFuture.java:1589)
                at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:1043)
                at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3370)
                at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3197)
                at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
                at java.base/java.lang.Thread.run(Unknown Source)
...
22-Aug-2022 21:39:50.823 SEVERE [ttl-cleanup-worker-#52%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.error JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to wait for initialization topology [err=Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=8280e085-0fdb-4288-9b4e-75779ac8e961, consistentId=**.**.**.**,127.0.0.1:47500, addrs=ArrayList [**.**.**.**, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, /**.**.**.**:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1661204387800, loc=false, ver=2.12.0#20220108-sha1:b1289f75, isClient=false], topic=TOPIC_CACHE, msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null, partsSizes=null, partHistCntrs=null, err=null, client=true, exchangeStartTime=1661204388092, finishMsg=null, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=535cb96f-3e0b-402d-bdcc-128446028ca5, consistentId=535cb96f-3e0b-402d-bdcc-128446028ca5, addrs=ArrayList [0:0:0:0:0:0:0:1, **.**.**.**, 127.0.0.1], sockAddrs=HashSet [/[0:0:0:0:0:0:0:1]:0, /127.0.0.1:0, /**.**.**.**:0], discPort=0, order=4, intOrder=0, lastExchangeTime=1661204387636, loc=true, ver=2.12.0#20220108-sha1:b1289f75, isClient=true], topVer=4, msgTemplate=null, span=o.a.i.i.processors.tracing.NoopSpan@1d6ea3e0, nodeId8=535cb96f, msg=null, type=NODE_JOINED, tstamp=1661204388076], nodeId=535cb96f, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1661204387076, nodeOrder=0], super=GridCacheMessage [msgId=1, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], err=null, skipPrepare=false]]], policy=2]]]]

Server Logs:

[23:46:41,726][SEVERE][exchange-worker-#50][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitions (rebalancing will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=d4c6c193-7f82-4ecd-83bc-65fe171ad51c, consistentId=10.30.161.13,127.0.0.1:47500, addrs=ArrayList [10.30.161.13, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, /10.30.161.13:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1661212000886, loc=true, ver=2.12.0#20220108-sha1:b1289f75, isClient=false], topVer=1, msgTemplate=null, span=o.a.i.i.processors.tracing.NoopSpan@cd2e57, nodeId8=d4c6c193, msg=null, type=NODE_JOINED, tstamp=1661212000630], nodeId=d4c6c193, evt=NODE_JOINED]
class org.apache.ignite.IgniteException: Failed to execute affinity operation on cache groups
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.forAllRegisteredCacheGroups(CacheAffinitySharedManager.java:1305)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.initCoordinatorCaches(CacheAffinitySharedManager.java:1974)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCoordinatorCaches(GridDhtPartitionsExchangeFuture.java:842)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:990)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3370)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3197)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$17.applyx(CacheAffinitySharedManager.java:2021)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$17.applyx(CacheAffinitySharedManager.java:1974)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.lambda$forAllRegisteredCacheGroups$e0a6939d$1(CacheAffinitySharedManager.java:1299)
        at org.apache.ignite.internal.util.IgniteUtils.lambda$null$1(IgniteUtils.java:11489)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        ... 1 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
[23:46:41,741][SEVERE][main][IgniteKernal] Got exception while starting (will rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to execute affinity operation on cache groups
        at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7774)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:234)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:152)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStart(GridCachePartitionExchangeManager.java:902)
        at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:724)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1376)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1798)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1720)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1159)
        at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1053)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:939)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:838)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:708)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:677)
        at org.apache.ignite.Ignition.start(Ignition.java:353)
        at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:359)
Caused by: class org.apache.ignite.IgniteException: Failed to execute affinity operation on cache groups
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.forAllRegisteredCacheGroups(CacheAffinitySharedManager.java:1305)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.initCoordinatorCaches(CacheAffinitySharedManager.java:1974)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCoordinatorCaches(GridDhtPartitionsExchangeFuture.java:842)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:990)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3370)
        at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3197)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$17.applyx(CacheAffinitySharedManager.java:2021)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$17.applyx(CacheAffinitySharedManager.java:1974)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.lambda$forAllRegisteredCacheGroups$e0a6939d$1(CacheAffinitySharedManager.java:1299)
        at org.apache.ignite.internal.util.IgniteUtils.lambda$null$1(IgniteUtils.java:11489)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        ... 1 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more
        Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to find completed exchange future to fetch affinity.
                ... 8 more

Edit:

I tried start fresh by:

  1. Deleting the ignite {WORK_DIR}.
  2. Deleting the Ingnite server files and reinstalling.

After this I no longer saw the Failed to find completed exchange future to fetch affinity but a very similar error I have seen in the past (see below). Normally this was fixed by adding the local port to the communication spi bean defenition on my client config, however that is currently there.

Local port config:

<bean id="communicationSpi" class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
    <property name="slowClientQueueLimit" value="1000" />
    <property name="localPort" value="47500"/>
</bean>

New client error:

23-Aug-2022 14:04:54.365 INFO [exchange-worker-#45%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.info Starting caches on local join performed in 958 ms.
23-Aug-2022 14:04:54.475 SEVERE [exchange-worker-#45%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.error Failed to send message to remote node [node=TcpDiscoveryNode [id=bfef6550-fe0d-424d-8761-703363fedb39, consistentId=10.30.161.13,127.0.0.1:47500, addrs=ArrayList [10.30.161.13, 127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500, /10.30.161.13:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1661263492635, loc=false, ver=2.12.0#20220108-sha1:b1289f75, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null, partsSizes=null, partHistCntrs=null, err=null, client=true, exchangeStartTime=1661263493070, finishMsg=null, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=89939261-6b12-44ec-bcef-7a36f2c3c85c, consistentId=89939261-6b12-44ec-bcef-7a36f2c3c85c, addrs=ArrayList [0:0:0:0:0:0:0:1, 10.30.161.13, 127.0.0.1], sockAddrs=HashSet [/[0:0:0:0:0:0:0:1]:0, /127.0.0.1:0, /10.30.161.13:0], discPort=0, order=2, intOrder=0, lastExchangeTime=1661263492344, loc=true, ver=2.12.0#20220108-sha1:b1289f75, isClient=true], topVer=2, msgTemplate=null, span=org.apache.ignite.internal.processors.tracing.NoopSpan@478e19db, nodeId8=89939261, msg=null, type=NODE_JOINED, tstamp=1661263492986], nodeId=89939261, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1661263491804, nodeOrder=0], super=GridCacheMessage [msgId=1, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], err=null, skipPrepare=false]]]]]
        class org.apache.ignite.IgniteCheckedException: No session found
                at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createNioSession(GridNioServerWrapper.java:660)
                at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:693)
                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:1264)
                at org.apache.ignite.spi.communication.tcp.internal.GridNioServerWrapper.createTcpClient(GridNioServerWrapper.java:691)
                at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.createCommunicationClient(ConnectionClientPool.java:481)
                at org.apache.ignite.spi.communication.tcp.internal.ConnectionClientPool.reserveClient(ConnectionClientPool.java:240)
                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1195)
                at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1142)
                at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2100)
                at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2193)
                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1266)
                at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendLocalPartitions(GridDhtPartitionsExchangeFuture.java:2201)
                at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.clientOnlyExchange(GridDhtPartitionsExchangeFuture.java:1589)
                at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:1043)
                at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3370)
                at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3197)
                at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
                at java.base/java.lang.Thread.run(Unknown Source)


Solution

  • To anyone else that finds this I had two errors in my config.

    1. IP finder only had a single port not range. Not sure how this changed. corrected bean:
    <bean id="ipFinder" class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
            <property name="shared" value="false" />
            <property name="addresses">
                <list>
                    <value>**.**.**.**:47500..47600</value>
                </list>
            </property>
    </bean>
    
    1. Somehow my server was not using the default com spi port number, not sure how this changed. Once I corrected these two issue it connected without any problem.