What is the proper way to manage large IStreams?

I am 开发者_如何学JAVAwriting a C# class library to transfer large volumes of data through COM automation using an IStream. It uses the CreateStreamOnHGlobal API call to create the stream, and the methods within System.Runtime.InteropServices.COMTypes.IStream to work with it.

My question is, when transferring large volumes of data, what is the best way to keep the memory footprint under control? Loading 100MB+ of file data into memory seems wasteful, and the client application would need to wait until that process is complete before downloading anything.

My plan was to create a reasonably sized stream and write to it multiple times. Before writing the next chunk of data, Seek back to the beginning and overwrite starting at the beginning. Am I going about this the right way, and is there a better method to solve this problem?

Consider using a file created with the attributes FILE_ATTRIBUTE_TEMPORARY and FILE_FLAG_DELETE_ON_CLOSE. Write you stuff in there. Windows will try to keep it in the disk cache unless memory runs out. It will self-destruct when you close the handle or when your program terminates (or crashes!). I learned about it here.

For temporary local storage of large amounts of data a memory mapped file is usually a good idea. It hands over the memory management responsibility to Windows which knows best how to do it. If you don't specify a file it will map into the paging file and never go to the physical disk unless it becomes necessary.

If you are using .NET 4.0 you got a managed API to memory mapped files.

Every stream implementation realies on buffer. Most of the time you will have 100% control over the buffer size. You can adjust buffer size to tune for performance vs. memory use balance. If you are not opening hundreds of stream at the same time I suggest be generous stream size as large as possible:

var readStream=new System.IO.File.OpenRead(sourceFilePath); var writeStream=new System.Io.File.Create(destinationFile);

byte[] buffer=new bytes[bufferSize]; int readLength; while((readLength=readStream.Read(0,0,bufferSize))>0) {
writeStream.Write(buffer,0,readLength); }

writeStream.Close(); readStream.Close();

When sending large messages using Windows Communication Foundation (WCF), it is often desirable to limit the amount of memory used to buffer those messages. One possible solution is to stream the message body (assuming the bulk of the data is in the body). However some protocols require buffering of the entire message. Reliable messaging and security are two such examples.

Another possible solution is to divide up the large message into smaller messages called chunks, send those chunks one chunk at a time, and reconstitute the large message on the receiving side. The application itself could do this chunking and de-chunking or it could use a custom channel to do it. The chunking channel sample shows how a custom protocol or layered channel can be used to do chunking and de-chunking of arbitrarily large messages.

Sample provided download available http://msdn.microsoft.com/en-us/library/aa717050.aspx