Mongoengine document consistency under concurrency
I am trying to create a dummy wrapper class ( a mongoengine Document ) that implements the interface of Redis hashes. For example:
class HashModel(mongoengine.Document):
'''
Represents a dictionary with a name.
Interface similar to Redis Hashes
'''
name = mongoengine.StringField()
adict = mongoengine.DictField()
def safe_reload(self): # because it fails if no object present
try: self.reload()
except:pass
def fix_key(self, key): # for mongoengine validation ( if you really read my code, please also answer why does mongoengine need that )
return key.replace(".","").replace("$","")
def hset(self, key, value):
self.safe_reload()
self.adict["%s" % self.fix_开发者_如何学Gokey(key)] = value
self.save(safe=True)
return True
def hexists(self, key):
self.safe_reload()
key = "%s" % self.fix_key(key)
return key in self.adict
Then, I use celery to execute some Tasks. Before that, I initialize a HashModel object on which the tasks perform some operations. But because of multiprocessing, I noticed some inconsistencies. Different processes "get" a different "snapshot" of the object, which is natural anyway. To bypass this issue, I re-initialize the object every time, so as to get an "almost" fresh snapshot every time.
Question: Is there a way to avoid re-initialization? Can I add some code, and what code, to my class above, to do that automatically?
EDIT: Answer: It seems, that the mongoengine.Document.reload() functions does this. I updated my code to show everything that is relevant to my next problem:
Back in the celery tasks, this problem emerged: When I hset(a_key, a_value), then later when I try to check if it exists, sometimes hexists(a_key) returns False. Even if I coerce it to hset, with:
while True:
self.handler.hset(a_key, a_value)
if self.handler.hexists(a_key):
break
later at some point (without deleting the a_key elsewhere),sometimes hexists(a_key) still yields False. How come that????
This is a pure race condition issue. My celery workers may simultaneously obtain an instance and perform different operations on it, then save it, no consistency anymore..
精彩评论