Add 'object' to a stack with expiry timers and get notified when it expires

I have a need to reserve an object (JSON) within my app for a period of time (typically 180 seconds) At some point the client may or may not come back and request this object by its key.

The tricky part is that I need to be notified when this object expires so I can return it to the available pool if the client hasn't already requested it.

The obvious solutions are to 开发者_开发百科use something like a timestamp in the database and then a periodic script to check for expired items but this doesn't feel like the nicest solution.

Ideally I'm looking for something like memcache that can call an event when an item expires, surely there is such a product out there?

My current framework is based around python, cherrpy, mongo, memcachce but I'm happy to add to it.

I would probably use threading.Timer for this. A Timer object will call a specified function with given arguments after a period of time. So write a function that returns a JSON object to the pool, and start a timer that specifies the specific JSON object that has been reserved. Additionally, you can cancel a timer before it fires, which you will want to do if the client actually requests the object before the reservation expires.

To keep track of the pool, I would probably use a dict where the JSON object is the key and the value is either None if the object is not checked out, or the Timer instance if it is checked out. A separate list could be used to keep track of what object should be given out next; pop() from the end of the list when taking an object out and append() it back on when it's returned. Beware of possible race conditions updating both of these structures!

Unless there are other processes that will be using this pool of objects, it doesn't seem there would be a reason to return the expired ones to the pool before you need to allocate a one. If you check for expired objects and return them to the pool before allocating new ones, that should cover the cleanup adequately.

In order to simplify the cleanup itself, consider creating a list of (expire_time, object) tuples and use bisect.insort() to keep the list ordered by the first value -- the expire time:

que_tuple = (time.time() + EXPIRY_DELAY, obj)
bisect.insort(my_list, que_tuple)

Each time you go to reserve another object, you can check for expired items to return to the pool. By using the sorted list, you only have to check the 0th element for being expired and then clean the elements in order until you find one that is not expired. Something like this:

time_to_act, obj = my_list[0]
while time.time() > time_to_act:
    return_to_pool( obj )   # whatever is necessary to clean-up your objects
    my_list.pop(0)
    time_to_act, obj = my_list[0]

If you do decide that you do need to implement a timer, this queue concept will still be useful. Otherwise, you could have a timer for every object's expiration. Often, there are a finite number of system timers available and you may find yourself running out of resources. The expiration queue (my_list) allows for a single timer to announce the next expiring object and the timer would be reset to the new head of the list.

I hope this helps!

I am almost sure that a solution around redis expire command should do the needful. It's simple to use. Here is example using Redis cli.

redis> HMSET myhash field1 "Hello" field2 "World"
OK
redis> HGET myhash field1
"Hello"
redis> EXPIRE myhash 180
(integer) 1
redis> HGET myhash field2
"World"
redis> HGET myhash field2 # after 180 secs
(nil)

Further Python has various redis client options. redis-natives-py provides annotation @temporary(after=None, at=None) .

Notification part

Redis wont notify you. May be application have logic to assume expiry after N seconds, and yes you can confirm that anytime with not expensive EXISTS command. Other option might be using greenlets to keep eye on time. Like it creates redis object, sleeps for N second and wakes up confirm with redis and notify. I am suggesting greenlet mainly becuase it is much more lightweight than a Python thread.