Ruby: start reading at arbitrary point in large file

I have some log files I would like to sift through. The content is exactly what you expect in a log file: many single lines of comma separated text. The files are about 4 gigs each. File.each_line or foreach takes about 20 minutes for one of them.

Since a simple foreach seems... simple (and slow), I was thinking that two separate threads might be able to work on the same file if I could only tell them where to start. But based on my (limited) knowledge, I can't decide if this is even possible.

Is there a way to start reading the file at a开发者_运维问答n arbitrary line?

To see what sort of difference slurping the entire file at once vs line-by-line, I tested against a file that is about 99MB, with over 1,000,000 lines.

greg-mbp-wireless:Desktop greg$ wc filelist.txt 
 1003002 1657573 99392863 filelist.txt

I put the following loop into a ruby file and ran it from the command line with the time command:

IO.read(ARGV.first).lines { |l|
}

greg-mbp-wireless:Desktop greg$ time ruby test.rb filelist.txt 

real    0m1.411s
user    0m0.653s
sys     0m0.169s

Then I changed it to read line-by-line and timed that too:

IO.readlines(ARGV.first) { |l|
}

greg-mbp-wireless:Desktop greg$ time ruby test.rb filelist.txt 

real    0m1.053s
user    0m0.741s
sys     0m0.278s

I'm not sure why, but reading line by line is faster. That might be tied to memory allocation as Ruby tries to load the entire file into RAM in the first example, or maybe it was an anomaly since I only did the test once for each file. Using a read with an explicit filesize might be faster as Ruby will know how much it's going to need to allocate in advance.

And that was all I needed to test this:

fcontent = ''
File.open(ARGV.first, 'r') do |fi|
  fsize = fi.size
  fcontent = fi.read(fsize)
end
puts fcontent.size

greg-mbp-wireless:Desktop greg$ time ruby test.rb filelist.txt 
99392863

real    0m0.168s
user    0m0.010s
sys     0m0.156s

Looks like knowing how much needs to be read makes quite a difference.

Adding back in the loop over the string buffer results in this:

File.open(ARGV.first, 'r') do |fi|
  fsize = fi.size
  fi.read(fsize).lines { |l| 
  }
end

greg-mbp-wireless:Desktop greg$ time ruby test.rb filelist.txt 

real    0m0.732s
user    0m0.572s
sys     0m0.158s

That's still an improvement.

If you used a Queue and fed it from a thread that was responsible for reading a file, then consumed the queue from whatever processes the incoming text then you might see a higher overall throughput.

If you want to start at a specific line in the file I would recommend just shelling out to tail.

excerpt = `tail -m +5000 filename.log`

This would give you the contents of filename.log from line 5000 to the end of the file.

For lines, it might be a bit difficult, but you can seek within a file to a certain byte.

IO#seek (link) and IO#pos (link) will both allow you to seek to a given byte within the file.

Try faster_csv if you haven't already and if thats still too slow use something that has native extensions in c like this - http://github.com/wwood/excelsior