How to develop a django->pylons file upload tool?

I have a Pylons application that provides web services to a Django-based front end. I need to be able to provide file upload and download to the users of the application.

Basically, they upload a file in the front end (which might be literally anything, and often quite large). The file is transmitted from client => django web server => pylons app server => internal file server.

The internal file server does not provide an API; the only way to put the file on it is by scp. So, no streaming is relevant on the pylons => fserv step. Moreover, the fserv cannot be accessed from the network tier that the django app lives on.

In the reverse process, the fserv provides an HTTP access method for its files, still inaccessible to the web server, so I need to connect a stream from fserv => pylons => django.

What does the Pylons side of this process look like? How can I minimize the latency of these file operations? I know I'm stuck on the write operation, given that I must send the whole file after receiving it.

Also, what does the Django side of this process look 开发者_运维百科like? How do I upload a file to a streaming upload service, and how do I download the same?

For the sake of minimizing argument, please assume that I am unable to change the main components of my system, and that firewalls and admin policy prevent my use of shared network resources to move files around.

You're not neccessarily stuck with scp'ing the uploaded file all at once. scp can take data from standard input, and in python you have fine control over input, output and error pipes of subprocesses.

For the Pylons part, I guess, the question could read, "How to do streaming HTTP uploads/downloads with Pylons"?

Streaming downloads, where Pylons controller serves large file (or acts as a proxy in front of fileserver), are easy. See "Streaming Content to Browser" on Pylons Wiki about this (the link is to google cache because wiki seems to be down ATM).

Streaming uploads in Pylons is more tricky, the main problem again being buffers in middlewares. There's an article about that as well on Pylons Wiki: "Hacking Pylons for handling large file upload" (google cache again).