Fast writing bytearray to file

I use _FileStream.Write(_ByteArray, 0, _ByteArray.Length); to write a bytearray to a file. I noticed that's very slow. I read a line from a text file, convert it to a bytearray and then need to write it to a new (large > 500 Mb) file. Please some advice t开发者_Go百科o speed up the write process.

FileStream.Write is basically what there is. It's possible that using a BufferedStream would help, but unlikely.

If you're really reading a single line of text which, when encoded, is 500MB then I wouldn't be surprised to find that most of the time is being spent performing encoding. You should be able to test that by doing the encoding and then throwing away the result.

Assuming the "encoding" you're performing is just Encoding.GetBytes(string), you might want to try using a StreamWriter to wrap the FileStream - it may work better by tricks like repeatedly encoding into the same array before writing to the file.

If you're actually reading a line at a time and appending that to the file, then:

• Obviously it's best if you keep both the input stream and output stream open throughout the operation. Don't repeatedly read and then write.

• You may get better performance using multiple threads or possibly asynchronous IO. This will partly depend on whether you're reading from and writing to the same drive.

• Using a StreamWriter is probably still a good idea.

Additionally when creating the file, you may want to look at using a constructor which accepts a FileOptions. Experiment with the available options, but I suspect you'll want SequentialScan and possibly WriteThrough.

If your writing nothing but Byte arrays, have you tried using BinaryWriter's Write method? Writing in bulk would probably also help with the speed. Perhaps you can read each line, convert the string to its bytes, store those bytes for a future write operation (i.e in a List or something), and every so often (after reading x lines) write a chunk to the disk.

BinaryWriter: http://msdn.microsoft.com/en-us/library/ms143302.aspx