Dealing with extremely large strings

Hey all, I am writing a logviewer application. These logs are between 100-200 megs (almost never larger than that) of pure text.

My application will be not only used to make the reading of said logs easier, but to offer tons of options on parsing (filters/searches, ect). Anywho, the files are on a network drive so I currently use FileStream and BufferedStream to read them in, which depending on the file takes about 15 seconds.

My first question is there a faster way to do this? ReadLine made parsing easier, but clocked in at about 25 seconds.

Secondly, if the user is just playing around with one file alot (using different combos of filters and searches, whats the best way to deal with it in the applications memory? I mean I could just re-read it each time but tha开发者_如何学Gots silly. I tried storing the whole file in a global StringBuilder which seemed to work, but I couldnt pass it on to other functions (OutOfMemoryException).

Whats the best way to deal with what ends up being a gigantic string? (Keep in mind no manipulations are being made, just pure reads/filter out what the user dosn't want to see.

Thanks guys!

Like Hans suggests, read the file in chunks and do it on demand when the user needs to move about the file. You can also try being smart and keeping nearby chunks in memory and discard distant chunks.

If you're doing this for fun, you might want to build a little in memory index as you process the file. Meaning for each line you process, keep some metadata about the line number, where in the file it starts, and what kind of line it is. That might make your options easier to work with.

Do you really have to read the entire file? Read only half of it. When the user is ready to look at another 'file' or set of dates, read the other half. Pick your own number here to find the best balance.