setCompressOutput in Hadoop

When should use and not to use FileOutputFormat.setCompressOutpu开发者_Python百科t(conf, true);?

I heard that it compresses mapper output. Is there any possibility to compress reducer side output?

(If my assumption is wrong, please clear me, how to compress mapper output and reducer output!)

You can control compression of the reducer output with mapred.output.compress, and compression of the mapper output with mapred.compress.map.output. These configuration keys can be set (to true or false) in the site-wide configuration file, in your job setup, or as -D options passed to Hadoop when you run your job.

Compressing map output is generally a good idea. I also compress reduce output when that output is not the final result, e.g. when I am running another job over the output of the previous job.

Compression often helps jobs finish faster (even though it requires extra processing for compression/decompression) because it can greatly decrease the amount of I/O.

You can pick compression codecs, too. We use LZO, which doesn't come with Hadoop but can be found here:

https://github.com/kevinweil/hadoop-lzo

LZO compresses pretty well with minimal CPU overhead. Bzip2 compresses very well, but with more significant overhead. Gzip compresses less well with moderate overhead. (These are generalizations.) I think LZO has the best balance of characteristics.