I have a use case where I want to have a globally distributed lock. We started out using SELECT .. FOR UPDATE, but that quickly started to have problems as we scaled up the number of servers. Also it didn't account for processes that checked out the lock and then died and failed to return the lock.
We need to be able to set an expiration on the lock (i.e. if the process who checked out the lock does not return it in 2 hours, the lock is automatically returned to the pool). I realize that this introduces the issue where we are ignoring locks, but we are fairly certain that the process has died if not complete in 2 hours. Also the job is idempotent, so if it is done more than once it's not a big deal.
I've looked through a number of distributed locking systems and come across this questions that have been extremely helpful. All of the solutions extend off of Java's java.util.concurrency.locks.Lock, which actually may be the issue I'm coming across because that interface doesn't have the expiration feature I need. We have a similar strategy to mongo-java-distributed-lock where we use MongoDB's findAndModify. We're considering:
as our distributed locking mechanism (all happen to implement java.util.concurrency.locks.Lock).
The biggest problem is that because java.util.concurrency.locks.Lock doesn't have an option for expiring a lock, these don't fit all the goals. This answer probably gets closest with hazelcast, but it is reliant on an entire server failing, not just a thread taking too long. Another option is possibly using a Samaphore with hazelcast as described here. I could have a reaper thread that is then able to cancel the locks of others if they are taking too long. With Mongo and Redis I could take advantage of their ability to expire objects, but that doesn't seem to be part of either of the libraries since they just implement java.util.concurrency.locks.Lock in the end.
So this was just a long winded way of asking, is there a distributed locking mechanism out there that I can have automatically expire after N seconds? Should I be looking at a different mechanism than java.util.concurrency.locks.Lock in this situation altogether?
You may use Redisson based on Redis server. It implements familiar Java data structures including java.util.Lock with distributed and scalable abilities. Including ability to setup lock release timeout. Usage example:
Config config = new Config();
// for single server
config.useSingleServer()
.setAddress("127.0.0.1:6379");
// or
// for master/slave servers
config.useSentinelConnection()
.setMasterName("mymaster")
.addSentinelAddress("127.0.0.1:26389", "127.0.0.1:26379");
Redisson redisson = Redisson.create(config);
Lock lock = redisson.getLock("anyLock");
try {
// unlock automatically after 10 seconds of hold
lock.lock(10, TimeUnit.SECONDS);
} finally {
lock.unlock();
}
...
redisson.shutdown();