What is the good configuration for Apache Ignite Grid computing with an OutOfMemoryError?

Question

I would like to use Grid computing functionnality of Apache Ignite, but I have a problem with the Java heap space (OutOfMemoryError) when I run my program on two nodes. To illustrate the problem, I propose a simple program which counts the number of words in a String :

try (Ignite ignite = Ignition.start("C:/Apache-Ignite/config/test-config.xml")) {
    IgniteCompute compute = ignite.compute();
    String[] elements = "Count characters using callable".repeat(10000).split(" ");
    Collection<IgniteCallable<Integer>> calls = new ArrayList<>();
    Arrays.stream(elements).forEach(word -> calls.add((IgniteCallable<Integer>) word::length));

    IgniteReducer<Integer, Integer> reducer = new IgniteReducer<>() {
        private Integer sum = 0;
        @Override
        public boolean collect(Integer integer) {
            sum += integer;
            return true;
        }
        @Override
        public Integer reduce() {
            return sum;
        }
    };

    IgniteFuture<Integer> result = compute.callAsync(calls, reducer);
    System.out.println("Result : " + result.get());
}

And the test-config.xml file :

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans
       http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="peerClassLoadingEnabled" value="true"/>
<property name="dataStorageConfiguration">
    <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
        <!-- Redefining the default region's settings -->
        <property name="defaultDataRegionConfiguration">
            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                <property name="name" value="Default_Region"/>
                <!-- Initial size. -->
                <property name="initialSize" value="#{2L * 1024 * 1024 * 1024}"/>
                <!-- Maximum size. -->
                <property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
            </bean>
        </property>
    </bean>
</property>
</bean>
</beans>

enter image description here
When I observe the heap during the progam execution, we can see clearly that the heap reachs his limit. Maybe the garbage collector is locked but I don't know.

In addition, if I run my program on my single local node (start with Ignition.start("...")), I don't have any problem.

Answer 1

I have just re-tried your case and it seems that callAsync() performance becomes progressively worse as calls.size() goes past 5000. In your case it is ~40000 so you're not going to see the completion of this call even if it does not cause OOM.

Please make sure to split your computes in batches of ~1000 calls. It is actually advised that compute job is substantial, in your case you are going to spend 99% of resources marshalling jobs and their arguments, anyway.

Nevertheless, I will file a ticket about this behavior.