Guaranteed memcached lock

So, I'm trying to implement a distributed lock using memcached and add()'s store only if does not exist contract (Java & spymemcached, but applicable in any language of course). Of course, if an instance goes away, then we lose the lock so the thought was add the lock 3 times (e.g. MyLock1, MyLock2, MyLock3) which will very likely hash out to 3 different instances.

But, I've realized that if an instances goes down the hash then obviously changes (using spymemcached's Redistribute failur开发者_开发技巧e mode) and so it's likely that when another attempt is made to add() the locks, that the hashes of all 3 lock's will not match any of the 2 remaining locks in the memcached cluster.

So...any other ideas for distributed locks using memcached? Or is it basically impossible to do a guaranteed lock like what I'm referring to?

EDIT: Ok, so in looking through spymemcached source code, for Redistribute mode, it simply goes to the next active memcached instance in it's list, rather than re-hashing anything, so it should work OK.

You can't, at least not reliably. memcached doesn't make any guarantees about data retention -- as a cache, it may discard data at any point without warning. Even if the memcache instance appears to have space available, it may have to evict data due to slab constraints.

If you need distributed locking, you'll need to look elsewhere -- memcached is the wrong tool for the job. For what it's worth, MySQL has locks: http://dev.mysql.com/doc/refman/5.1/en/miscellaneous-functions.html

If you really want to use memcached to avoid introducing more stuff/complexity into your environment, then consider a very small but dedicated memcached config just for locking.

But if you're open to solutions that don't rely on memcached, then personally I'd use zookeeper to implement a distributed lock in java. I'd also use the Netflix curator utils to make this easier.

If you're using Java, I recommend using Hazelcast (1.9+), it does support distributed locks across a Cluster, and is easy to create one.

Hazelcast guarantees that if a server that holds a lock goes down, the lock will be released.

http://hazelcast.com/docs/1.9.4/manual/single_html/#Lock

Also Hazelcast exposes the same contract as a memcached, so if you need to access it out of the JVM you can do it (this sample demonstrates that any client would work):

Java:

MemcachedClient client = new MemcachedClient(AddrUtil.getAddresses("10.20.17.1:5701 10.20.17.2:5701"));
client.set("key1", 3600, "value1");
System.out.println(client.get("key1"));

PHP:

<?php
    $memcache = new Memcache;
    $memcache->connect('10.20.17.1', 5701) or die ("Could not connect");
    $memcache->set('key1','value1',0,3600);
    $get_result = $memcache->get('key1'); //retrieve your data
    var_dump($get_result); //show it
?>

Documentation of 1.9+: http://hazelcast.com/docs/1.9.4/manual/single_html/

Hope it helps.