I am trying to use Ignite as an in-memory database for key-value pairs with values ranging from 50MB to 800MB. It looks like Ignite allocates heap space through the JVM that it never cleans up, even if the cache entries are off heap, are cleared, there are no connected clients and no running operations. My machine would not be able to handle such memory consumption, hence I am looking for a way to clean up some memory.
My test scenario is as follows:
I am testing Ignite locally through a python script with the pyignite thin client:
client = Client()
client.connect('localhost', 10800)
my_cache = client.get_or_create_cache('default')
my_cache.clear()
data = createData() #creates 800 000 000 bytes test data
def put(caches):
i = caches
while i > 0:
my_cache.put('my key' + str(i), data)
i -= 1
put(5)
my_cache.remove_all()
my_cache.destroy()
client.close()
The script writes the 800 MB data sequentially in 5 different cache entries. The following snapshot illustrates how Ignite's Heap grows to its peak point, which by itself is understandable, but afterwards stays around 10GB even after performing GC:
Performing a second test run with the same data does not result in bigger heap consumption after GC, which leads me to believe that Ignite allocates buffers for the incoming data internally, which correspond with the size of the data. This memory consumption is simply too great and my machine would not be able to deal with it in the long run.
The ignite configuration is quite simple:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Durable memory configuration. -->
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="dataRegionConfigurations">
<list>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<!-- Custom region name. -->
<property name="name" value="10GB_Region"/>
<!-- 100 MB initial size. -->
<property name="initialSize" value="#{100L * 1024 * 1024}"/>
<!-- 10GB maximum size. -->
<property name="maxSize" value="#{10096L * 1024 * 1024}"/>
</bean>
</list>
</property>
<!-- Redefining the default region's settings -->
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<property name="maxSize" value="#{5L * 1024 * 1024 * 1024}"/>
</bean>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<!-- Partitioned cache example configuration (Atomic mode). -->
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="default"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="1"/>
<property name="dataRegionName" value="10GB_Region"/>
</bean>
</list>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<value>127.0.0.1:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
Any help will be very much appreciated!
My apologies for the long delay to anyone who is interested, but I did get an answer to my question after contacting the Ignite mailing list. Apparently Ignite was never meant to consume such large amounts of data in a single query. The described behavior is expect and has to do with dividing the data into packages and sending it to other nodes or putting it into persistence. The suggested solution is to chunk the data into much smaller bits. Since this is currently not possible in my case, I had let go of Ignite and look for other approaches to store my data.