Available implementations of a distributed REST queue

I have the following (common, I guess) scenario: a master node holding a list of items (urls, jobs, whatever) to be processed, and a set of N worker nodes.

Each worker pops an item from the queue, does something, then notifies the master node that the job has successfully finished.

A worker may push new items to the master node, for these to be queued for processing.

Requirements are quite simple:

• no job gets executed twice
• no job gets picked by two nodes (ie. "pop" is atomic)
• if a job fails, after a fixed timeout, the job is again available for processing
• the number of concurrent workers is potentially big
• several workers may live on a single node
• master and workers are not assumed to be in the same network
• pop and push must be exposed as a REST API (ie. the queue is language-agnostic)
• items must be stored persistently on the master node (ie. no in-memory solutions)

I am not able to find simple and lightweight REST implementations: I have looked at RabbitMQ, Celery, Google App Engine and a bunch of other less mature projects, but all of them seem quite complex to manage, and a bit like overkill for what I need.

Any solution that I might 开发者_Python百科be overlooking?

Amazon SQS might be what you want: http://aws.amazon.com/sqs/

Jan

I went through the same search. I found celery was close (also looked at others like octobot) - none seemed as simple as I desired, and were missing a few things. I found celery was fairly simple, however introduces a reasonable amount of dependencies which I didn't already have in the mix, so I went with something bespoke instead (erlang based)