spring boot Hazelcast configuration for integration tests

We use JCache and provide a backend/CacheManager for the JCache specification using the following cache configuration. Hazelcast is autoconfigured by spring and hence we do not need to provide the CacheManager explicitly, but only provide our hazlecast configuration.

@Configuration
public class CacheConfig {
    public static final int TTL_INFINITE = 0;

    @Bean
    public Config hazelCastConfig() {
        // do not allow hazelcast to send data to hazelcast, see
        // http://docs.hazelcast.org/docs/latest-development/manual/html/Preface/Phone_Home.html
        GroupProperty.PHONE_HOME_ENABLED.setSystemProperty("false");
        return new Config()
                .setInstanceName("hazelcast-instance")
                // create a cache
                .addCacheConfig(new CacheSimpleConfig()
                        .setName(RateLimiterServiceImpl.CACHE_NAME))
                // store it distributed
                .addMapConfig(new MapConfig()
                        .setName(RateLimiterServiceImpl.CACHE_NAME)
                        .setTimeToLiveSeconds(RateLimiterServiceImpl.CACHE_SECONDS_TO_LIVE)
                        .setEvictionPolicy(EvictionPolicy.LFU))
                // create a cache
                .addCacheConfig(new CacheSimpleConfig()
                        .setName(I18nServiceImpl.CACHE_NAME))
                // store it distributed
                .addMapConfig(new MapConfig()
                        .setName(I18nServiceImpl.CACHE_NAME)
                        .setTimeToLiveSeconds(I18nServiceImpl.CACHE_SECONDS_TO_LIVE)
                        .setEvictionPolicy(EvictionPolicy.LRU));
    }

}

In production and when running tests locally, everything is fine. But using gitlab CI we get the following error during integration tests:

java.lang.IllegalStateException: null
        at com.hazelcast.cache.impl.AbstractHazelcastCacheManager.checkIfManagerNotClosed(AbstractHazelcastCacheManager.java:374) ~[hazelcast-3.9.2.jar:3.9.2]

and with hazelcast 3.10.5

java.lang.IllegalStateException: CacheManager /hz/ is already closed.
    at com.hazelcast.cache.impl.AbstractHazelcastCacheManager.ensureOpen(AbstractHazelcastCacheManager.java:366) ~[hazelcast-3.10.5.jar:3.10.5]
    at com.hazelcast.cache.impl.AbstractHazelcastCacheManager.getCache(AbstractHazelcastCacheManager.java:219) ~[hazelcast-3.10.5.jar:3.10.5]
    at com.hazelcast.cache.impl.AbstractHazelcastCacheManager.getCache(AbstractHazelcastCacheManager.java:67) ~[hazelcast-3.10.5.jar:3.10.5]
    at org.springframework.cache.jcache.JCacheCacheManager.getMissingCache(JCacheCacheManager.java:114) ~[spring-context-support-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.cache.support.AbstractCacheManager.getCache(AbstractCacheManager.java:97) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]

This is the test failing:

mockMvc.perform(put("/translations/{locale}", locale)
                    .contentType(MediaType.APPLICATION_JSON_UTF8)
                    .content(dto)
                    .andExpect(status().isNoContent());
// gives a 500 with the above error message

How should we configure the integration tests to work with hazelcast?

Solution

CacheManager is and iterface, when using CacheManager with hazelcast, it is uses HazelcastCacheManagerImpl.

When connection to hazelcast is dropped or lost for some reason, exception, hazelcast is shutdown etc. internal hazelcast connection used by HazelcastCacheManager becomes closed and as far whole CacheManager becomes closed, attempt to get particular Cache from such cache manager and perform r/w operation under such cache will lead to "java.lang.IllegalStateException: CacheManager /hz/ is already closed." exception.

When hazelcast is back to life again (reconnected or else) you need to manually reconnect CacheManager to hazelcast. I played with 3.5.7 version of hazelcast and to solve this issue we had created CacheManager as a proxy bean.

In case we detect CacheManager has lost connection to real hz, we start attempts to manually reconnect CacheManager and in case success, replace links to CacheManager with new one.

This is a question how to re-init spring beans on runtime.

I assume in your test you have similar situation, during test connection to hz is lost, cache becomes closed. Even if you restart hazelcast, but not reinitialize CacheManager bean - it may point to old closed hz connection and will not reconnect automatically.

If you use @SpringBootTest mechanism, using annotation @DirtiesContext may help so every test will re-init whole spring context so if in previous test you have destroyed hazelcast connection, restart hz or else, in next test DirtiesContext will force CacheManger re-initialization and help to have repeatable tests results.