Web services + Celery

I plan to use celery to process incoming web service requests. I understand that celery is used mostly to process asynchronous tasks. However celery has lot of features that I like and could benefit from in my project - priorities, rate limits, distributed architecture etc.

I am just struggling with the design. I would like to have web service that creates and starts the task that will call subtasks. Original task needs results from the subtasks and then when original task is finished I return result back to the client through web service. I know I could call tasks synchronou开发者_Python百科sly but that it is not a good practice.

Thanks,

The scatter/gather thing looks like it could be a map/reduce job. If the mapreduce part is important to you, go with a specialised framework like Disco or Hadoop. Otherwise, you need some kind of completion signal, so that you can fire a reply to the user once all subtasks are done or cancelled. For example, a counter of how many subtasks are yet to terminate. The subtask that brings the counter to zero can push a new reply task that pushes a reply to the user and closes the circle.

Look at Mongrel2, an asynchronous web framework, for an example of this kind of circular request path.