I have written a file processing program and now it needs to read from a zipped file(.gz unzipped file may get as large as 2TB),

Is there a sed equivalent for zipped files like (zcat/cat) or else what would be the best approach to do the following efficiently

    ONE=`zcat filename.gz| sed -n $counts`

$counts : coun开发者_StackOverflow中文版ter to read(line by line)

The above method works, but is quite slow for large file as I need to read each line and perform the matching on certain fields.

Thanks

EDIT

Though not directly helpful, here are a set of zcommands

http://www.cyberciti.biz/tips/decompress-and-expand-text-files.html

Well you either can have more speed (i.e. use uncompressed files) or more free space (i.e. use compressed files and the pipe you showed)... sorry. Using compressed files will always have an overhead.

If you understand the internal structure of the compression format it is possible that you could write a pattern matcher that can operate on compressed data without fully decompressing it, but instead by simply determining from the compressed data if the pattern would be present in a given piece of decompressed data.

If the pattern has any complexity at all this sounds like quite a complicated project as you'd have to handle cases where the pattern could be satisfied by the combination of output from two (or more) separate pieces of decompression.