So, I'm trying to implement a distributed lock using memcached and add()'s store only if does not exist contract (Java & spymemcached, but applicable in any language of course). Of course, if an instance goes away, then we lose the lock so the thought was add the lock 3 times (e.g. MyLock1, MyLock2, MyLock3) which will very likely hash out to 3 different instances.
But, I've realized that if an instances goes down the hash then obviously changes (using spymemcached's Redistribute failure mode) and so it's likely that when another attempt is made to add() the locks, that the hashes of all 3 lock's will not match any of the 2 remaining locks in the memcached cluster.
So...any other ideas for distributed locks using memcached? Or is it basically impossible to do a guaranteed lock like what I'm referring to?
EDIT: Ok, so in looking through spymemcached source code, for Redistribute mode, it simply goes to the next active memcached instance in it's list, rather than re-hashing anything, so it should work OK.
If you really want to use memcached to avoid introducing more stuff/complexity into your environment, then consider a very small but dedicated memcached config just for locking.
But if you're open to solutions that don't rely on memcached, then personally I'd use zookeeper to implement a distributed lock in java. I'd also use the Netflix curator utils to make this easier.