low priority http upload in .net

I'm writing a program that uploads huge amounts of data and I need to limit it's interference with web browsing and other user activities.

The upload is composed of many large-ish files that are transferred individually, the connection must be a standard HTTP POST (I have no control of the server) and I need to control the HTTP headers (the server uses them for authentication and metadata)

It's important that the upload will resume full speed when the user is no longer using the internet because otherwise it will never finish (I expect it will need to run for a week or more at full speed to complete).

I want to solve this problem by somehow making my HTTP connection low priority, detecting open browser windows and slowing down does not solve开发者_如何学JAVA the problem because (a) the user may be using a non-browser app (FTP, twitter client, e-mail, etc.) and (b) I don't want to slow down if there's an open idle web browser window.

I've found BITS but I think it's not relevant for me since I need it to be a standard HTTP POST.

I'm using .net 3.5, the program is written in C# and I'm currently using HttpWebRequest for the upload.

Clarification: I’m writing consumer software that will run on the customer’s personal computer at home. My beta testers complain that the internet is slow when they run my program (understandable, since I am using all their bandwidth) so I want to give higher priority to other programs so their internet is no longer slow.

There is no fancy network infrastructure that can prioritize packets on the network and no IT team to install and configure anything, I do expect most customers will have a cheap wireless router they got for free from their ISP

Simultaneously keep track of number of bytes your app sends and the total bytes sent on the network using the System.Net.NetworkInformation.IPv4InterfaceStatistics class' bytesSent Property at a given interval. Subtract the total bytes your app has sent in that interval from the total bytes sent on the network (during the same interval). If the difference is high enough to where you need to throttle your uploading then do so. Once the difference becomes small enough, crank up the uploading.

I hope you have the throttling end of the problem covered, so you only need to know two things:

• WHEN to reduce upload speed

and

• HOW MUCH to reduce it

Now, using other API-s is maybe fine, but why learn them and introduce all new kind of bug generators, when you can do something more simple.

Now - the problem with uploads with most internet connections (say ADSL) is that they consume precious UPLOAD that is extremely limited, say 256-512k. User probably report the problems because your upload is killing the requests they send, so they can't get any data back.

I am proposing that you do two things... to resolve WHEN problem, do this:

• create simple GET request for your web server
• measure time that is needed for the server to fully process and respond, by reading the response to the end
• use one cut-off value for 'busy' and 'free' connection, or measure times for each client and calculate their cut-off value
• if time is greater, you are busy, do throttling, else, go full speed
• you can repeat this as much as you can - for example every minute

To resolve HOW MUCH is another question. First, you should measure how much you can upload when nobody else is using the connection - your baseline upload speed per client. Then, using the WHEN technique, you can throttle down the upload until you get 'free' result that will work for their connection.

Hope I made myself clear - ask if anything needs clarification.

EDIT:

Few more thing to consider:

1. sending binary data through the POST is wasteful since everything is encoded in BASE64. You should inspect some other means of upload.
2. throttling against the web server could be problem because if you upload big chunk of data slowly, maybe server will timeout
3. if you combine 1+2, you could devise a plan that can splice the data into chunks, say 1KB or more, and send them one by one. It alone could solve your throttling problem, if you do it in serialized fashion.

You might consider taking a more passive approach to the problem. You simply need to set the packets your application generate to a lower priority than the other packets origination from the system. Then let the network take care of scheduling the packets.

It has been my experience that even low end consumer network devices (Personal firewalls, gateways, cable modems, DSL modems, even the Windows OS's, etc.) support traffic shaping. Use this capability and set the traffic you generate to a low priority and all other packets will default to a medium priority if their priority is not explicitly set. The medium packets will be allotted the bulk of the available bandwidth when they exist and your triffic will be squeezed out. When the medium priority packets do not exist on the network your packets will be allowed to use the available bandwidth.

Since I haven't tried any of this myself, I can only provide pointers.

For detecting network activity, you can try using NMAPI or Packet Monitor

WinPCap also has a ported .Net library but not sure if it captures http traffic.

Other than these, there are no QoS libraries in .Net AFAIK.

If you're willing to delve into C++, QoS API can help.

None of this is straighforward and looking at your comments, I'm not sure if this is feasible within your timeline, but it is what it is. Sorry!

Thinking outside the box :

What about just scanning for human activity : keyboard and mouse action, or presence of screensaver, or locked screen ? Then when no user is in front of the computer, you upload full speed, otherwise, upload at a minimum rate.

Im not sure how to implemnt it in .NET but couldnt you perhaps do like smaller test with round-trip-time (RTT) to the server and if you like the result you get back you increase the package load. If you have "congestion" on you client then you should perhaps have a little longer RTT and then lower the load?

I'd go for a solution where you get the maximum upload speed for the machine and 'poll' every 5 seconds if you're application is not uploading if there is any network traffic.

This should make sure you don't eat all network traffic but have the maximum speed most of the time.

Is providing a bittorrent-like maximum bandwidth cap out of the question? You could adjust the cap based on the time of day (full speed in the middle of the night, 50% during peak hours, for example), or give the user the option to configure their maximum speed and scheduling options.