Need CGI (or another solution compatible with IIS 7) to handle *massive* uploads

We need to handle massive file uploads without spending resources on an IIS 7 server. To emphasize how light-weight this needs to be, let's say that we need to handle file uploads of sizes that are completely insane, like 100GB uploads, or something that can continue running for an extremely long time without consuming additional resources. Basically we need something that gives us control over the reception of the file from the moment it starts to the moment it ends.

A bit of background:

We're using ColdFusion as the server-side processor, but it has failed us when handling uploads beyond about 1GB and we've exhausted our configuration options. There's a long story behind that, but essentially, if a .cfm page (ColdFusion) is the destination of the file upload and it goes over about 1GB, it gives a 503 error... even if the target file doesn't exist. So clearly too much is going on merely by telling the server that we intend to process the file with a .cfm page.

We suspect that this is due to Java limitations because the server (or really, the workstation in this case) does not show any signs of load on CPU or memory. Since we have limited memory and this website is intended for a lot of concurrent uploads, we can't trust simply raising the virtual machine memory usage, especially because that simply doesn't work currently, even for a single connection... let alone the hundreds of concurrent connections we expect when we go live.

So we're down to writing a specialized solution using CGI that will handle file uploads only. Basically, we need control on the server-side that we don't get with ColdFusion or ASP.NET because those technologies do so many things on their own, behind the scenes, without giving us the control we need. They always end up spending up too many resources one way or the other for an arguably obvious reason; what we're trying to do is completely insane and not the intended function of those technologies. That's why we want a specialized uploader through CGI that bypasses all that ColdFusion/ASP.NET magic that keeps getting in the way, hoping it gives us the control we need.

But before we spent countless hours on this, I figured I'd ask around and see if anyone knows of a proper solution to开发者_StackOverflow中文版 this problem that might be viable in our case.

The only real restriction here is that it has to be CGI, and it has to run on IIS 7, therefore a Windows "Server" environment. We're fine with it being written in Python, Perl, name it... provided it can run as a CGI, but it has to run as a CGI... unless of course someone has better ideas on how to do this.

So the magic question is; are there CGI solutions out there that already do this or are we stuck with writing it on our own, hoping that the reason no one else has done it already is some other than it being impossible?

Thanks in advance.

You're not going to get reliable multi-GB uploads from a dumb client (eg a browser and standard upload behaviour). Been there, done that, written commercial digital asset management solutions handling huge files.

The key to any degree of reliability in this scenario is chunking - you need to be able to chunk the upload, send each chunk as a discrete file, and re-assemble it server side.

What are your client restrictions though (if any)? Can you use a java applet? Could you even have a client side app ?

One possible starting point for a browser based solution would be the jupload opensource project but there are plenty of others.

You want WebDAV, not CGI. It provides all the nice bits that make file transfers not suck, like resuming and pausing.

the windows TCP stack is limited to 4GB file uploads. Anymore than that is not possible.