Hazelcast AtomicLong Data loss When multiple member left

Question

Hazelcast fails when multiple members disconnect from the cluster. My scenario is so basic and my configuration has 3 bakcup option(it does not work). I have 4 members in a cluster and i use AtomicLong API to save my key->value. When all members are alive, everything is perfect. However, some data loss occurs when i kill 2 members at the same time(without waiting for a while). My member counts are always 4. is there any way to avoid this kind of data loss?

  Config config = new Config();
  NetworkConfig network = config.getNetworkConfig();
  network.setPort(DistributedCacheData.getInstance().getPort());
  config.getCacheConfig("default").setBackupCount(3);
  JoinConfig join = network.getJoin();
  join.getMulticastConfig().setEnabled(false);
  join.getTcpIpConfig().setEnabled(true);  
  config.setNetworkConfig(network);
  config.setInstanceName("member-name-here");

Thanks.

Answer 1

IAtomicLong has hard-coded 1 sync backup, you cannot configure it to have more than 1 backup. What you are doing is configuring Cache with 3 backups.

Below is a example demonstrates multiple member disconnect for IMap

Config config = new Config();
config.getMapConfig("myMap").setBackupCount(3);
HazelcastInstance[] instances = {
        Hazelcast.newHazelcastInstance(config),
        Hazelcast.newHazelcastInstance(config),
        Hazelcast.newHazelcastInstance(config),
        Hazelcast.newHazelcastInstance(config)
};
IMap<Integer, Integer> myMap = instances[0].getMap("myMap");
for (int i = 0; i < 1000; i++) {
    myMap.set(i, i);
}
System.out.println(myMap.size());
instances[1].getLifecycleService().terminate();
instances[2].getLifecycleService().terminate();
System.out.println(myMap.size());