How to set OFF-HEAP or ON-HEAP Memory in Apache Ignite?

I have an 8-node cluster, each having 16GB RAM. Previously, I have used Ignite 1.2 version without any problem. Recently I moved to 2.6. Now, for a small dataset, it works fine. But for a large dataset, each time it is giving me an error "out of memory" or "Failed to send a message" etc. My Ignite configuration for each node is below:

<property name="defaultDataRegionConfiguration">
    <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
         <property name="initialSize" value="#{500 * 1024 * 1024}"/>
         <property name="maxSize" value="#{12L * 1024 * 1024 * 1024}"/>
         <property name="pageEvictionMode" value="RANDOM_2_LRU"/>
         <property name="persistenceEnabled" value="true"/>                               
         <property name="metricsEnabled" value="true"/>
    </bean>
</property>

And I ran Ignite in each node using "Ignite.sh -Xmx15g -Xms15g". I have configured two caches(cache1, cache2) as below and run a join query on them.

RendezvousAffinityFunction affFunc = new RendezvousAffinityFunction();        
affFunc.setExcludeNeighbors(true);              
affFunc.setPartitions(1024);

CacheConfiguration<Long, EMP> cache1 = new CacheConfiguration<>(EMP_CACHE);
cache1.setIndexedTypes(Long.class, EMP.class);
cache1.setSqlFunctionClasses(EMPFunctions.class);
cache1.setCacheMode(CacheMode.PARTITIONED);
cache1.setAffinity(affFunc);

My largest dataset contains more than 4million records. I tried with many ways, even with the on-heap option. But join operation with large dataset gives me an error. But for a smaller dataset, it's just working fine. Anyone help me to configure ON-HEAP or OFF-HEAP for my cluster.

Solution

In the current memory architecture (Apache Ignite 2.x, see this link) you can't choose on-heap memory only.

The data is always stored on the off-heap. It will be fetched (although not completely) onto the heap to perform all kinds of processing. For example, while JOIN and WHERE of an SQL query can mostly be done right in the off-heap, the final result set must be fetched onto the heap.

In your case, you're committing more memory to Ignire than you have. You commit 12GB off-heap + 15GB heap = 27GB, which is obviously more than your 16 GB RAM.

In general, you should commit just enough memory to store the data (+ indexes and overhead!) you need on the off-heap, and you can give the rest to the heap. You should also leave some RAM free for the system needs. But with persistence enabled you can have even more data than off-heap memory - but keep in mind that if your data doesn't fit in the off-heap data region, performance will suffer.

By the way, remove your pageEvictionMode - it has no meaning when persistence is enabled (yes, documentation fails to highlight that).

Finally, if you get an OutOfMemory for your heap space, it probably means that your SQL result set is too large. To workaround that you can either

make the result set smaller by splitting your SQL query into several ones returning less data
increase heap size
use SqlFieldsQuery.setLazy(true) - with this flag Ignite will try to split the result set into chunks, if possible, and load them onto heap one by one