开发者

hadoop : 1 map multiple reducers with each reducer having different functionality? possible?

Here is an example:

Is it possible to have same mapper run against multiple reducers at the same time? like

map output : {1:[1,2,3,4,5,4,3,2], 4:[5,4,6,7,8,9,5,3,3,2], 3:[1,5,4,3,5,6,7,8,9,1], so on}  

reducer1 : sum of all numbers  
reducer2 : average of all numbers  
reducer3 : mode of all numbers   

act on the the same key like  

reducer1 output: {1:sum of values, 2:sum of values, and so on}  
reducer2 output: {1:开发者_JAVA百科avg of values, 2: avg of values and so on}
reducer3 output: {1:mode of values, 2: mode of values, and so on}  

and so on..Please let me know.


I really wanted to answer this for you but it's already been asked. Hadoop one Map and multiple Reduce


No it is not possible. But you can implement your own reducer that will calculate all the stuff for you and output it.

You can make an custom writable for that.


http://www.cloudera.com/blog/2011/04/simple-moving-average-secondary-sort-and-mapreduce-part-3/

This actually does pretty much what you are looking for.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜