Hi we are using Apache Ignite 2.7 (8 nodes, 120GB each) and configuring 16GB heap and a 100GB data region (with persistence on). Using native memory tracking we see the usually expected categories like heap, thread etc. are as expected but "Internal" (i.e. off-heap) is a whopping 132GB. That is on top of everything else the JVM needs to run. With such a huge memory request by the JVM the system is being driven into out of memory conditions (OS out of RAM).
As an experiment we reduced the data region to 1GB and measured JVM internal memory use before and after grid activation (grid is being activated by a client node we attach). We saw Internal (read: unsafe off-heap) memory jump from 62,154 to 32,897,187 KB on grid activation. So the 32GB overhead seems to be regardless of the size of the data region.
This 32GB of extra system RAM usage is a real problem for us. Why is Ignite doing this and how to we control it?
Thanks
Here is a typical native memory summary we are seeings. Note the HUGE Internal allocation.
native memory Total: reserved=156688325KB, committed=156439245KB - Java Heap (reserved=16777216KB, committed=16777216KB) (mmap: reserved=16777216KB, committed=16777216KB) - Class (reserved=112257KB, committed=111489KB) (classes #17951) (malloc=1665KB #17624) (mmap: reserved=110592KB, committed=109824KB) - Thread (reserved=229015KB, committed=229015KB) (thread #223) (stack: reserved=228032KB, committed=228032KB) (malloc=723KB #1128) (arena=260KB #432) - Code (reserved=255790KB, committed=40250KB) (malloc=6190KB #11547) (mmap: reserved=249600KB, committed=34060KB) - GC (reserved=704014KB, committed=704014KB) (malloc=48654KB #22251) (mmap: reserved=655360KB, committed=655360KB) - Compiler (reserved=420KB, committed=420KB) (malloc=289KB #1284) (arena=131KB #15) - Internal (reserved=138544815KB, committed=138544811KB) (malloc=138544779KB #35177) (mmap: reserved=36KB, committed=32KB) - Symbol (reserved=26536KB, committed=26536KB) (malloc=24002KB #216741) (arena=2533KB #1) - Native Memory Tracking (reserved=4822KB, committed=4822KB) (malloc=30KB #346) (tracking overhead=4791KB) - Arena Chunk (reserved=673KB, committed=673KB) (malloc=673KB) - Unknown (reserved=32768KB, committed=0KB) (mmap: reserved=32768KB, committed=0KB)
PS
We have the default data region set to 128MB, the systemRegionMaxSize set to 8GB and systemRegionInitialSize set to 512MB.
Config:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="gridLogger">
<bean class="org.apache.ignite.logger.log4j.Log4JLogger">
<constructor-arg type="java.lang.String" value="/opt/ignite/apache-ignite/config/log4j.xml"/>
</bean>
</property>
<property name="metricsLogFrequency" value="600000"/>
<property name="rebalanceThreadPoolSize" value="12"/>
<property name="peerClassLoadingEnabled" value="true"/>
<property name="publicThreadPoolSize" value="32"/>
<property name="systemThreadPoolSize" value="32"/>
<property name="workDirectory" value="/data/ignite/work"/>
<property name="segmentationPolicy" value="RESTART_JVM"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="checkpointReadLockTimeout" value="0"/>
<property name="systemRegionInitialSize" value="#{512L * 1024 * 1024}"/>
<property name="systemRegionMaxSize" value="#{8L * 1024 * 1024 * 1024}"/>
<property name="storagePath" value="/data/ignite/persistentStore"/>
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<property name="initialSize" value="67108864"/>
<property name="maxSize" value="134217728"/>
<property name="persistenceEnabled" value="false"/>
<property name="metricsEnabled" value="true"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Tiered_Region"/>
<property name="initialSize" value="53687091200"/>
<property name="maxSize" value="53687091200"/>
<property name="persistenceEnabled" value="true"/>
<property name="pageEvictionMode" value="RANDOM_2_LRU"/>
<property name="evictionThreshold" value="0.75"/>
<property name="metricsEnabled" value="true"/>
</bean>
</list>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="default"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="0"/>
</bean>
</list>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="messageQueueLimit" value="#{1 * 1024}"/>
<property name="idleConnectionTimeout" value="30000"/>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
<property name="awsCredentials" ref="aws.creds"/>
<property name="bucketName" value="project-test-xyz"/>
</bean>
</property>
</bean>
</property>
</bean>
<bean id="aws.creds" class="com.amazonaws.auth.BasicAWSCredentials">
<constructor-arg value="foo"/>
<constructor-arg value="bar"/>
</bean>
</beans>
[ Adding logs below ]
[2019-05-17 22:28:39,592][WARN ][main][IgniteKernal] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2019-05-17 22:28:39,593][WARN ][main][IgniteKernal] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments. [2019-05-17 22:28:40,141][WARN ][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) [2019-05-17 22:28:40,214][WARN ][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2019-05-17 22:28:41,690][WARN ][main][GridCacheDatabaseSharedManager] DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files [2019-05-17 22:28:41,826][WARN ][main][PartitionsEvictManager] Logging at INFO level without checking if INFO level is enabled: Evict partition permits=4 [2019-05-17 22:28:46,291][WARN ][main][IgniteKernal] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=12516MB, available=14008MB] log4j: Finalizing appender named [null]. [2019-05-17 22:31:19,958][WARN ][disco-event-worker-#42][GridDiscoveryManager] Local node's value of 'java.net.preferIPv4Stack' system property differs from remote node's (all nodes in topology should have identical value) [locPreferIpV4=null, rmtPreferIpV4=true, locId8=f25228c0, rmtId8=eac4211d, rmtAddrs=[192.168.1.5/127.0.0.1, /192.168.1.5], rmtNode=ClusterNode [id=eac4211d-c272-4eb0-9bd5-f91dfa34a0e9, order=2, addr=[127.0.0.1, 192.168.1.5], daemon=false]] [2019-05-17 22:32:24,265][WARN ][exchange-worker-#43][GridAffinityAssignmentCache] Logging at INFO level without checking if INFO level is enabled: Local node affinity assignment distribution is not ideal [cache=default, expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00, actualBackups=0, warningThreshold=50.00%] [2019-05-17 22:32:24,269][WARN ][exchange-worker-#43][GridAffinityAssignmentCache] Logging at INFO level without checking if INFO level is enabled: Local node affinity assignment distribution is not ideal [cache=default, expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00, actualBackups=0, warningThreshold=50.00%] [2019-05-17 22:32:24,850][WARN ][exchange-worker-#43][GridAffinityAssignmentCache] Logging at INFO level without checking if INFO level is enabled: Local node affinity assignment distribution is not ideal [cache=default, expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00, actualBackups=0, warningThreshold=50.00%] [2019-05-17 22:32:24,911][WARN ][disco-notifier-worker-#41][GridClusterStateProcessor] Logging at INFO level without checking if INFO level is enabled: Received state change finish message: true 22:33:49.086 [exchange-worker-#43] INFO c.b.aa.ceres.loader.S3CacheLoader - load eb5445c7-d7fa-4018-95b6-63c4a0911eae received inject ignite instance IgniteKernal [longJVMPauseDetector=LongJVMPauseDetector [workerRef=Thread[jvm-pause-detector-worker,5,main], longPausesCnt=0, longPausesTotalDuration=0, longPausesTimestamps=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], longPausesDurations=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], cfg=IgniteConfiguration [igniteInstanceName=null, pubPoolSize=32, svcPoolSize=32, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=16, mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/opt/ignite/apache-ignite, igniteWorkDir=/data/ignite/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e, nodeId=f25228c0-afbc-4626-990a-68f97fd5b258, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@44a3f602], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=NOOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=60000, commSpi=TcpCommunicationSpi [connectGate=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$ConnectGateway@6020964a, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@3f2874d5, enableForcibleNodeKill=false, enableTroubleshootingLog=true, locAddr=null, locHost=0.0.0.0/0.0.0.0, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=GridNioServer [selectorSpins=0, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@7873ad1, directMode=true], GridConnectionBytesVerifyFilter], closed=false, directBuf=true, tcpNoDelay=true, sockSndBuf=32768, sockRcvBuf=32768, writeTimeout=2000, idleTimeout=600000, skipWrite=false, skipRead=false, locAddr=0.0.0.0/0.0.0.0:47100, order=LITTLE_ENDIAN, sndQueueLimit=1024, directMode=true, sslFilter=null, msgQueueLsnr=null, readerMoveCnt=0, writerMoveCnt=0, readWriteSelectorsAssign=false], shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=47100, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@7b757828[Count = 0], stopping=false], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@282cb7c7, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@50de186c, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@5a3bc7ed, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=60000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=1024, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=Default_Region, maxSize=134217728, initSize=67108864, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=true, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=[DataRegionConfiguration [name=Tiered_Region, maxSize=8589934592, initSize=8589934592, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=true, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0]], storagePath=/data/ignite/persistentStore, checkpointFreq=180000, lockWaitTime=30000, checkpointThreads=8, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@2fb68ec6, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=true, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null], igniteInstanceName=null, startTime=1558132126418, rsrcCtx=org.apache.ignite.internal.processors.resource.GridSpringResourceContextImpl@556d0e12, reconnectState=ReconnectState [firstReconnectFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1426466647], curReconnectFut=null, reconnectDone=null]]
I thik that's Checkpoint Page Buffer, which is 20% of data region's size by default.
You may specify it explicitly to make sure you don't forget it, and decrease the region size accordingly to make sure you don't run out of RAM.
Should only be applicable to persistent regions.
Note that you should also expect your OS take a few GBs towards its data structures and block caches, so I don't think you should allocate 116G of 120G to Ignite's Off-Heap. Don't forget about Heap either.