shared variable in map reduce

I need a variable that shared between reduce tasks and each of reduce tasks ca开发者_运维百科n read and write on it atomically. The reason that I need such a variable is to give a unique identifier to each files that created by reduce task (number of files which created by reduce tasks is not deterministic).

Thanks

In my understanding ZooKeeper is specially built to maintain atomic access to the cluster wide variables.

I would recommend using FileSystem.createNewFile().

Have a look here:

http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/fs/FileSystem.html#createNewFile%28org.apache.hadoop.fs.Path%29

All the outout files produced by the reducers already have unique names part-r-00001 and such. There is a partition number you can read in case you need that number from your code.

Centralized counters that must be guaranteed unique break a lot of the scalability of Hadoop.

So if you need something different then I would use something like a Sha1 of the task id of the reducer to get something that is unique over multiple jobs.