Search code examples
javamemorymemory-leaksignite

Apache Ignite Fabric 2.1 crashes on start


I am new to Apache ignite and trying to run it on my local machine.While executing bat file for Ignite,i am getting error related to Out Of Memory.How can i resolve it?.Ignite launches but crashes after works due to Message Queue limit issues.Where would i have to make a change to resolve this issue.Your help will be highly appreciated.Log file for reference is attached below.

[03:36:09,797][INFO][main][IgniteKernal] 

>>>    __________  ________________  
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7    // /  / / / _/    
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 2.1.0#20170720-sha1:a6ca5c8a
>>> 2017 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org

[03:36:09,797][INFO][main][IgniteKernal] Config URL: file:/C:/apache-ignite-fabric-2.1.0-bin/config/default-config.xml
[03:36:09,797][INFO][main][IgniteKernal] Daemon mode: off
[03:36:09,797][INFO][main][IgniteKernal] OS: Windows 10 10.0 x86
[03:36:09,797][INFO][main][IgniteKernal] OS user: Administrator
[03:36:09,797][INFO][main][IgniteKernal] PID: 11180
[03:36:09,797][INFO][main][IgniteKernal] Language runtime: Java Platform API Specification ver. 1.8
[03:36:09,797][INFO][main][IgniteKernal] VM information: Java(TM) SE Runtime Environment 1.8.0_131-b11 Oracle Corporation Java HotSpot(TM) Server VM 25.131-b11
[03:36:09,797][INFO][main][IgniteKernal] VM total memory: 0.96GB
[03:36:09,797][INFO][main][IgniteKernal] Remote Management [restart: on, REST: on, JMX (remote: on, port: 49112, auth: off, ssl: off)]
[03:36:09,797][INFO][main][IgniteKernal] IGNITE_HOME=C:\apache-ignite-fabric-2.1.0-bin
[03:36:09,797][INFO][main][IgniteKernal] VM arguments: [-Xms1g, -Xmx1g, -XX:+AggressiveOpts, -XX:MaxMetaspaceSize=256m, -DIGNITE_QUIET=true, -DIGNITE_SUCCESS_FILE=C:\apache-ignite-fabric-2.1.0-bin\work\ignite_success_1230775c-566a-421b-b7d8-5c64e068779a, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=49112, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -DIGNITE_HOME=C:\apache-ignite-fabric-2.1.0-bin, -DIGNITE_PROG_NAME=ignite.bat]
[03:36:09,797][INFO][main][IgniteKernal] System cache's MemoryPolicy size is configured to 40 MB. Use MemoryConfiguration.systemCacheMemorySize property to change the setting.
[03:36:09,797][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc' memoryPolicy: ['ignite-sys-cache']]
[03:36:09,812][INFO][main][IgniteKernal] 3-rd party licenses can be found at: C:\apache-ignite-fabric-2.1.0-bin\libs\licenses
[03:36:09,812][WARNING][pub-#11%null%][GridDiagnostic] This operating system has been tested less rigorously: Windows 10 10.0 x86. Our team will appreciate the feedback if you experience any problems running ignite in this environment.
[03:36:09,875][INFO][main][IgnitePluginProcessor] Configured plugins:
[03:36:09,875][INFO][main][IgnitePluginProcessor]   ^-- None
[03:36:09,875][INFO][main][IgnitePluginProcessor] 
[03:36:09,932][INFO][main][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false]
[03:36:20,174][WARNING][main][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[03:36:20,189][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
[03:36:20,220][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival).
[03:36:20,220][INFO][main][IgniteKernal] Security status [authentication=off, tls/ssl=off]
[03:36:20,455][INFO][main][SqlListenerProcessor] SQL connector processor has started on TCP port 10800
[03:36:20,495][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[03:36:21,117][INFO][main][IgniteKernal] Non-loopback local IPs: 192.168.1.12, 192.168.141.1, 192.168.255.1, 2001:0:4137:9e76:3801:b15:d8cd:8e1a, fe80:0:0:0:0:5efe:c0a8:10c%net11, fe80:0:0:0:0:5efe:c0a8:8d01%net17, fe80:0:0:0:0:5efe:c0a8:ff01%net18, fe80:0:0:0:14c1:3be7:bd23:2d53%eth1, fe80:0:0:0:20c7:34ab:8407:ae6e%eth7, fe80:0:0:0:2801:549e:2f77:5a8d%eth9, fe80:0:0:0:3801:b15:d8cd:8e1a%net9, fe80:0:0:0:5d31:9e33:861e:2c86%wlan1, fe80:0:0:0:b8d4:adda:5e30:d8d8%net6, fe80:0:0:0:f8e2:aff9:73f7:d1f3%wlan0
[03:36:21,117][INFO][main][IgniteKernal] Enabled local MACs: 00000000000000E0, 005056C00001, 005056C00008, 889E935B5E5C, 9453302064E0, 9653302064E0, C85B7652D25B
[03:36:21,148][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=204d6b27-acc2-4d4a-9a91-451d41cb1611]
[03:36:21,148][WARNING][main][TcpDiscoveryMulticastIpFinder] TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is recommended in production to specify at least one address in TcpDiscoveryMulticastIpFinder.getAddresses() configuration property)
[03:36:24,711][INFO][exchange-worker-#34%null%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], crd=true, evt=10, node=TcpDiscoveryNode [id=204d6b27-acc2-4d4a-9a91-451d41cb1611, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.1.12, 192.168.141.1, 192.168.255.1, 2001:0:4137:9e76:3801:b15:d8cd:8e1a], sockAddrs=[/192.168.1.12:47500, NISBCOM005301.mobilink.net.pk/192.168.255.1:47500, /192.168.141.1:47500, /2001:0:4137:9e76:3801:b15:d8cd:8e1a:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1502490981148, loc=true, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=false], evtNode=TcpDiscoveryNode [id=204d6b27-acc2-4d4a-9a91-451d41cb1611, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.1.12, 192.168.141.1, 192.168.255.1, 2001:0:4137:9e76:3801:b15:d8cd:8e1a], sockAddrs=[/192.168.1.12:47500, NISBCOM005301.mobilink.net.pk/192.168.255.1:47500, /192.168.141.1:47500, /2001:0:4137:9e76:3801:b15:d8cd:8e1a:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1502490981148, loc=true, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=false], customEvt=null]
[03:36:24,711][WARNING][exchange-worker-#34%null%][IgniteCacheDatabaseSharedManager] No user-defined default MemoryPolicy found; system default of 1GB size will be used.
[03:36:24,726][SEVERE][exchange-worker-#34%null%][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], nodeId=204d6b27, evt=NODE_JOINED]
java.lang.OutOfMemoryError
    at sun.misc.Unsafe.allocateMemory(Native Method)
    at org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1054)
    at org.apache.ignite.internal.mem.unsafe.UnsafeMemoryProvider.nextRegion(UnsafeMemoryProvider.java:80)
    at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.addSegment(PageMemoryNoStoreImpl.java:610)
    at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.start(PageMemoryNoStoreImpl.java:230)
    at org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.startMemoryPolicies(IgniteCacheDatabaseSharedManager.java:194)
    at org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.onActivate(IgniteCacheDatabaseSharedManager.java:949)
    at org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:244)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:602)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
    at java.lang.Thread.run(Thread.java:748)
[03:36:24,726][INFO][exchange-worker-#34%null%][GridDhtPartitionsExchangeFuture] Snapshot initialization completed [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], time=0ms]
[03:36:24,726][SEVERE][main][IgniteKernal] Got exception while starting (will rollback startup routine).
java.lang.OutOfMemoryError
    at sun.misc.Unsafe.allocateMemory(Native Method)
    at org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1054)
    at org.apache.ignite.internal.mem.unsafe.UnsafeMemoryProvider.nextRegion(UnsafeMemoryProvider.java:80)
    at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.addSegment(PageMemoryNoStoreImpl.java:610)
    at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.start(PageMemoryNoStoreImpl.java:230)
    at org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.startMemoryPolicies(IgniteCacheDatabaseSharedManager.java:194)
    at org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.onActivate(IgniteCacheDatabaseSharedManager.java:949)
    at org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:244)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:602)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
    at java.lang.Thread.run(Thread.java:748)
[03:36:24,726][SEVERE][exchange-worker-#34%null%][GridCachePartitionExchangeManager] Runtime error caught during grid runnable execution: GridWorker [name=partition-exchanger, igniteInstanceName=null, finished=false, hashCode=7687550, interrupted=false, runner=exchange-worker-#34%null%]
java.lang.OutOfMemoryError
    at sun.misc.Unsafe.allocateMemory(Native Method)
    at org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1054)
    at org.apache.ignite.internal.mem.unsafe.UnsafeMemoryProvider.nextRegion(UnsafeMemoryProvider.java:80)
    at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.addSegment(PageMemoryNoStoreImpl.java:610)
    at org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.start(PageMemoryNoStoreImpl.java:230)
    at org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.startMemoryPolicies(IgniteCacheDatabaseSharedManager.java:194)
    at org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager.onActivate(IgniteCacheDatabaseSharedManager.java:949)
    at org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:244)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:602)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
    at java.lang.Thread.run(Thread.java:748)
[03:36:24,726][INFO][main][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary
[03:36:24,845][INFO][main][IgniteKernal] 

>>> +---------------------------------------------------------------------------------+
>>> Ignite ver. 2.1.0#20170720-sha1:a6ca5c8a97e9a4c9d73d40ce76d1504c14ba1940 stopped OK
>>> +---------------------------------------------------------------------------------+
>>> Grid uptime: 00:00:15:477

Solution

  • by default Ignite trying to allocate 0.8 of all physical memory on machine for it pageMemory. So, it looks like in this case, after starting Ignite with 1gb heap, on your machine left less than 80% of free memory.

    It's not a good idea to define smaller heap size, I would recommend to configure MemoryPolicy for nodes to override default value for pageMemory and make memory allocation lesser.

    Check this documentation for additional information and examples