Wait for the result of an external webservice inside a task

I need to write a bunch of tasks where every task needs to query an external web service. The web service always replies with a 202 ACCEPTED status and points in the Location header to the URI where the result can be polled. The time it takes for this web service to deliver the result can vary from 2 seconds to a minu开发者_StackOverflow社区te. I was wondering whats the best approach to program my celery task. For now I send the request and start a while loop till I successfully poll the result, eg:

while True:
    result = poll_webservice()
    if result:
        break
    else:
        time.sleep(5)

[ continue with the rest of the task ]

While this certainly works it seems very crude to me and also I block the celery worker till the result is polled. Is there any better approach?

You' re surely killing your resources. Just query your external web service and save the URI to be polled (use a cache or a db).Then you can have a periodic task that collects the results, if ready...

I'd be thinking about a celery task that polls the Location provided by the 202 response. If it is completed, then it processes it, else it re-queues itself (or a copy of itself) for some time later.

For bonus points, if you have lots of these tasks, you could increase the time between polls each time the response is not ready.

How about using task.retry ?

@task(max_retries=100)
def poll_webservice_task(url):
    result = poll_webservice(url)
    if result:
       return result
    poll_webservice_task.retry(countdown=5)