App Engine TimeOut Error: Serving a third-party API with an image stored on App Engine

I'm building an application in Python on App Engine. My application receives images as email attachments. When an email comes in, I grab the image and need to send it to a third party API.

The first thing I did was:

1) make a POST request to the third party API with the image data

I stopped this method because I had some pretty bad encoding problems with urllib2 and a MultipartPostHandler.

The second thing I'm doing right now is

1) Put the image in the incoming email in the Datastore

2) Put it in the memcache

3) Send to the API an URL that serves the image (using the memcache or, if not found in the memcache, the Datastore)

The problem I read on my logs is: DeadlineExceededError: ApplicationError: 5

More precisely, I see two requests in my logs:

- first, the incoming email

- then, the third party API HTTP call to my image on the URL I gave him

The incoming email ends up with the DeadlineExceededError. The third party API call to my application ends up fine, serving correctly the image.

My interpretation:

It looks like App Engine waits for a response from the third party API, then closes because of a timeout, and then serves the request made by the third party API for the image. Unfortunately, as the connection is closed, I cannot get the useful information provided by the third party API once it has received my image.

My questions:

1) Can App Engine handle a request from a host it supposes to get a response of?

2) If not, how can I bypass this problem?

If you directly use the App Engine URLfetch API, you can adjust the timeout for your request. The default is 5 seconds, and it can be increased to 10 seconds for normal handlers, or to 10 minutes for fetches within task queue tasks or cron jobs.

If the external API is going to take more than 10 seconds to respond, probably your best bet would be to have your email handler fire off a task that calls the API with a very high timeout set (although almost certainly it would be better to fix your "pretty bad encoding problems"; how bad can encoding binary data to POST be?)

To answer your first question: if you're using dev_appserver, no, you can't handle any requests at all while you've got an external request pending; dev_appserver is single-threaded and handles 1 request at a time. The production environment should be able to scale to do this; however, if you have handlers that are waiting 10 seconds for a urlfetch, the scheduler might not scale your application well since the latency of incoming requests is one of the factors in auto-scaling.