Path MTU Discovery

I am developing an application which processes (video processing, etc) and sends large files (up to tens of gigabytes) over a network. I am sending the files using FTP. To improve the performance/memory consumption of the application, I would like optimize the buffers so that i dont send packets which are too large and get fragmented. The problem i have is that i dont have much RAM to hold the file data whilst sending. Basically, i read just enough bytes from disk, process it and immediately send to the destination. Currently i am looking to implement MTU path discovery.

I am familiar with the basic concept of how to do it. I would like to know开发者_Python百科 if there is any .NET API in windows which keeps track of the MTU to a destination ?

I am guessing there is no such thing, but a friend of mine told me windows vista keeps track.

I am developing this application for windows XP, but i would like to learn if there is such a network tracking API in windows.

winsock doesn't support reporting the discovered MTU, even though other TCP/IP stacks do (e.g. AIX through a socket option IP_GETPMTU). As winsock cannot report that, .NET can't provide an API (which would have to be on top of winsock).

I recommend sending data in chunks of 64kiB. This is the maximum IP packet size, and likely larger than the MTU, so the stack will send several full segments. The last fragment may be smaller, but then, the system may delay sending it (because it still needs to receive acknowledges for earlier data), so if you follow up quickly with the next send of 64kiB, the system will combine the chunks again into packets using the path mtu.

This is not something you have to do yourself, provided you're using TCP. Also, there is no relation to the size of your send buffers. There can be multiple packets in transit (up to the TCP window size), but they will no longer be in the buffer you maintain.

If you are using UDP, you might care about fragmentation, but if so, why are you implementing your own protocol to transfer large files? Use TCP instead and don't worry about it. If you are implementing your own protocol over UDP with congestion control and all the other things you need, then set the DF bit on the packets and deal with the responses.

If you are using TCP, you don't need to worry about fragmentation at all, the stack does that for you.

Use FtpWebRequest and FtpWebResponse. The matter is these classes don't use huge buffers but streams (which implemented using best sized buffer). The memory usage will be very minimal.

The packet size is not under your control but under network driver.

If you need best speed then implement data send/receive thru TcpClient class on both server and client sides.

UPD: Here is a sample on how files should be uploaded: http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx See class AsynchronousFtpUpLoader.

One more thing. Some time ago I was experimenting with MTU. All my changes decreased the data transfer speed. The windows network driver is smart enough to use best MTU.

PMTU Discovery is used by you're TCP stack solely for optimizing packet size when sending. At you're level you only see the stream (ie. before packetization even takes place, let alone fragmentation). What you're problem really sounds like is you are trying to send data as you can, however you're tcp/ip connection is slower, so this is causing the space needed in ram to build up. Since you're using FTP, I would expect the FTP implementation you are using to already support and understand this??? Since you are asking for buffers, it sounds like a roll you're own based on a socket or something like that. If you are using the synchronous send methods, when the socket buffer is full, it should block you're sends until it has buffer space.

Can you maybe give more details on what API's you're using to send the file, Builtin vs. roll you're own? Whether it be socket, networkstream, etc?