I've been looking into using a local entry listener instead of a normal entry listener so that an event is only processes by a single listener.
I've found various posts on this topic such as this, this, this, this and this. It seems that a local entry listener is indeed the way to go for handling an event only once in a multi node cluster.
However, I'm not sure how such a local entry listener would function under failure conditions. For instance, what happens to an evicted-event if the node which is the master for that entry is unavailable. Will the backup pick this up in time? Or could the event be missed due to hazelcast needing some time to figure out the master is down and a new master should be elected? Is this different between the older AP-system and the new CP-subsystem?
We've refrained from using a local entry listener. Instead we are now using the executorservice from hazelcast to schedule a named task. In this way, we can correctly respond to changes in the cluster. It does seem like hazelcast has a preferred member on which a task is executed, but that isn't an issue for us.