java - Hadoop: passing data between consecutive mapreduce tasks -

I have macarized jobs in the following ways: map1 -> less 1 -> map 2 -> less 2 during map1 As a side step effect, I only calculate the required data during phase 2, so I do not want to pass it completely from the series What is the best way to pass this data, so that at least 2 steps But can I get data from Map 2 and Map 1?

Thanks

Depending on your comments, A and b . Then, to go to the 2Redgers to go to A reducer 1 and b , with the output of the mapper. The best option I can see is to:

JOB 1:
A to differentiate b , Use MultipleOutputs in the first job ... Use a normal prefix (for example in the values) type B the intermediate output of Mapper 1, which they A will be different from the output in Reducer 1, when you see the prefix, delete it and type b s into the b output path.

Job 2:
Use multiple input in your second job Use Mapper 2 for input which this process does And an IdentityMapper for b will only forward it to spread again <2> B2/2 2, where you will also process the output of Mapper 2.

A simple code snippet:

  MultipleInputs .addInputPath (conf, new path ("/ input / path / off / job / 2"), sequence file input format. Clash, mapper 2. class); MultipleInputs.addInputPath (conf, new path ("/ path / k / b"), sequence file input format. Category, IdentityMapper.class); Conf.setReducerClass (Reducer2.class);  

Where is multiple input import org.apache.hadoop.mapred.lib.MultipleInputs; .

You can not obtain data in Reduce 2 and process them in the same way on which you can process the mapper 2, unless you use the mapper for them. Generally, you can not use a reducer without a mapper, using an IdentityMapper closest to it.

If you want to process b in any other way, you can get them through the distributed cache, or it's a number or two, just this value Set a value with conf.set ("variableName", variable variable); using conf.get ("variableName", defaultValue), . . . in the method. .