Concurrent threads in GemFire CacheWriter

Question

We are currently using Cassandra as NoSQL Database and GemFire as In memory Database. We have been using the GemFire CacheWriter to insert the records in Cassandra. I would like your feedback on whether it’s a good engineering practice to use Concurrent threads in CacheWriter to insert/Update records. Your feedback on this would be appreciated.

public class GenericWriter<K, V> extends CacheWriterAdapter<K, V> implements Declarable {

    private static Logger log = LoggerFactory.getLogger(GenericWriter.class);

    @Autowired
    private CassandraOperations cassandraOperations;

    ExecutorService executor = null;

    @Override
    public void beforeCreate(EntryEvent<K, V> e) {

    executor = Executors.newSingleThreadExecutor();

        executor.submit(() -> {
            if (eventOperation.equals("CREATE") || eventOperation.equalsIgnoreCase("PUTALL_CREATE")) {
                try {
                    cassandraOperations.insert(e.getNewValue());
                } catch (CassandraConnectionFailureException | CassandraWriteTimeoutException
                        | CassandraInternalException cassException) {
                } catch (Exception ex) {
                    log.error("Exception in GenericCacheWriter->" + ExceptionUtils.getStackTrace(ex));
                    throw ex;
                }
            }
        });
        executor.shutdown();
    }

    @Override
    public void init(Properties arg0) {
        // TODO Auto-generated method stub

    }
}

Answer 1

Yes, it's a fine pattern but remove the Executor and partition your data such that all updates into GemFire go to one and only one node. Partition Cassandra the same way. Put a write lock around the Cassandra update. Use this only when your throughput is low.

If you need high throughput, use the AsyncEventListener and guarantee eventual consistency to your users. If you must use Executors in the AEL, use them in a way so as to throw an exception in the main thread. If the update fails after a number of tries, you write the failed entry to a different region with an expiration of a few seconds or a minute. When that expires, retry the operation. Keep doing this until the succeeds and then and only then, delete the expired entry.

You will need to track version numbers and what you are updating watching old values/ new values if order of updates is important to you or not.