How does EHCache using Terracotta handle eviction from the distributed heap?

Question

We've recently started using EHCache with Terracotta to run a distributed data cache for application data. Let's say that any client node has about 2gb for their heaps, whereas server nodes have 8gb. We generate a lot of data, about 1.5gb a day.

Generally, any one client will be using a specific day's dataset (of about 1.5gb), but the server obviously has to hold all of them.

The way I'd like expiration to work is on an LRU basis, when the heap grows to big. So if any particular L1 client side cache gets too big (say, switching from day1 to day2) I'd expect it to evict from L1 all of the day1 data. If L2 gets too big as we get the 6th dataset, then the oldest dataset gets expired completely. I don't really have any opinions about what time-to-live or time-to-idle values should be, so I'm leaving them unset.

After a few days of looking at it, I don't think this is working as I'd expect. For instance, I ran a test with a L2 max elements of 4. I populated it with four elements. Then I went to put a fifth element. Cache.put() returned without exceptions, but an immediately following Cache.get() with the same key returned null!

So my question is, how do I get EHCache+Terracotta to do what I want, or at least something close?

Answer 1

After much messing around, it turns out the mysterious behavior is determined by whether or not the L1 cache has any knowledge of what's in the L2 cache. The attached unit tests are meant to be executed in pairs, each element of the pair run in a separate JVM to avoid EHCache singleton-ness.

The way I think a well-behaved L1->L2 relationship should work is that the if you do a .put() without error, you should be able to do a get() of the same key immediately after without a problem (assuming no other concurrently running threads are messing with stuff). However, with Terracotta EHCache, if that .put() requires something to be evicted, the eviction does NOT happen, and the put() is silently ignored, unless the L1 client 'knows' about the keys that can be evicted. In the *JVM2 tests, it finds out about those other keys by using .getAllWithLoader(), and then the *JVM2 tests work as expected.

So what we have done to address our original is to make it so that periodically clients do a .getAllWithLoader(). Then we can be sure that all the eviction rules are followed.

package com.mine;

import java.io.Serializable;

import junit.framework.TestCase;
import net.sf.ehcache.Cache;
import net.sf.ehcache.CacheManager;
import net.sf.ehcache.Element;
import net.sf.ehcache.config.CacheConfiguration;
import net.sf.ehcache.config.Configuration;
import net.sf.ehcache.config.TerracottaClientConfiguration;
import net.sf.ehcache.config.TerracottaConfiguration;

public class CacheEvictionTest extends TestCase {

    private static final String SERVER = "localhost:9510";
    private CacheManager cacheManager;
    private Cache cache;
    private final Serializable keyA = "a";
    private final Serializable keyB = "b";
    private final Serializable keyC = "c";

    @Override
    protected void setUp() throws Exception {
        Configuration configuration = new Configuration();
        TerracottaClientConfiguration terracottaConfig = new TerracottaClientConfiguration();
        terracottaConfig.setUrl(SERVER);
        configuration.addTerracottaConfig(terracottaConfig);

        int maxElementsInMemory = 1;
        int maxElementsOnDisk = 2;
        long timeToIdleSeconds = 15;
        long timeToLiveSeconds = 15;
        String cacheName = "TEST_CACHE";
        CacheConfiguration amoebaCache = new CacheConfiguration(cacheName, maxElementsInMemory).statistics(true)
                        .terracotta(new TerracottaConfiguration())
                        .logging(true)
                        .maxElementsOnDisk(maxElementsOnDisk)
                        .timeToIdleSeconds(timeToIdleSeconds)
                        .timeToLiveSeconds(timeToLiveSeconds);
        configuration.addCache(amoebaCache);
        configuration.addDefaultCache(new CacheConfiguration("default", 0));

        cacheManager = new CacheManager(configuration);
        cache = cacheManager.getCache(cacheName);
    }

    @Override
    protected void tearDown() throws Exception {
        if (cache != null) {
            cache.removeAll();
            cache.clearStatistics();
        }
    }

    public void testMaxElementOnDiskEvictionJVM1() throws Exception {
        cache.clearStatistics();

        cache.put(new Element(keyA, keyA));
        cache.put(new Element(keyB, keyB));

        cache = null;
    }

    public void testMaxElementOnDiskEvictionJVM2() throws Exception {
        assertEquals(2, cache.getSize());

        for (Object key : cache.getKeys()) {
            cache.get(key;
        }

        cache.put(new Element(keyC, keyC));

        assertEquals(2, cache.getSize());
        assertNotNull(cache.get(keyC));
    }

    public void testEvictsExpiredElementsFromDiskWhenNotInMemoryAndWeNeverKnewAboutItJVM1() throws Exception {
        cache.clearStatistics();
        cache.put(new Element(keyA, keyA));

        cache = null;
        cacheManager = null;
    }

    public void testEvictsExpiredElementsFromDiskWhenNotInMemoryAndWeNeverKnewAboutItJVM2() throws Exception {
        cache.clearStatistics();

        for (Object key : cache.getKeys()) {
            cache.get(key;
        }
        assertEquals(0, cache.getStatistics().getEvictionCount());

        Thread.sleep(20000);

        assertEquals(1, cache.getStatistics().getEvictionCount());
    }
}