Batch I/O performance problem [closed]

Closed. This question is off-topic. It is not currently accepting answers.

---

Want to improve this question? Update the question so it's on-topic for Stack Overflow.

Closed 11 years ago.

Improve this question

I have a c#.net application that has to process a big number of files. The processing of this files is done through command line calls to external tools. Every one of this files goes through 4 processes which create a new file that feeds the next process in a pipeline fashion. As a result, we have 5 files (the original + the result of the 4 processes) pe开发者_高级运维r every original file.

I am processing now through a c# generated batch file that I execute through command line. The I/O takes too long. I have been sugested to use a RAM Disk to speed the intermediate result I/O processes. But I have read (in this page) that this is stale technology.

What solution is there to speed up the processing in this case?

What good freeware solution is there if I want to give the RAM Disk a try? I am using Windows 7.

Thanks Oscar

---

Right, RAM disks went the way of the dodo when Windows acquired a file system cache. You didn't include any relevant info about the programs and files so guessing is required. If the files are large then the file system cache is probably not big enough to be able to store all the file data. The typical diagnostic is the programs completing quickly at first but then suddenly taking a long time when the cache is storing too much data waiting to be written to the disk. The cure for that is a 64-bit operating system with lots of RAM.

If the files are very small then the processing time would be dominated by each process getting created and initialized before it can pump the data. This is unlikely to be the problem since this normally doesn't take more than a couple of dozen milliseconds. But a program, say, checking for an update for itself by contacting an Internet server isn't unusual.

Which leaves the likeliest explanation: disk I/O is the bottleneck. It cannot make the reading of the first file any faster, the data has to come off the disk and that runs at glacial hard disk speeds. You need a faster disk. SSDs are nice.

We don't really know until you measure.