hadoop : 1 map multiple reducers with each reducer having different functionality? possible?
Here is an example:
Is it possible to have same mapper run against multiple reducers at the same time? like
map output : {1:[1,2,3,4,5,4,3,2], 4:[5,4,6,7,8,9,5,3,3,2], 3:[1,5,4,3,5,6,7,8,9,1], so on}
reducer1 : sum of all numbers
reducer2 : average of all numbers
reducer3 : mode of all numbers
act on the the same key like
reducer1 output: {1:sum of values, 2:sum of values, and so on}
reducer2 output: {1:开发者_JAVA百科avg of values, 2: avg of values and so on}
reducer3 output: {1:mode of values, 2: mode of values, and so on}
and so on..Please let me know.
I really wanted to answer this for you but it's already been asked. Hadoop one Map and multiple Reduce
No it is not possible. But you can implement your own reducer that will calculate all the stuff for you and output it.
You can make an custom writable for that.
http://www.cloudera.com/blog/2011/04/simple-moving-average-secondary-sort-and-mapreduce-part-3/
This actually does pretty much what you are looking for.
精彩评论