从map中键入不匹配的值：expected org.apache.hadoop.io.NullWritable，recieved org.apache.hadoop.io.Text

我正在尝试调整现有问题以满足我的需求..

基本上输入是简单的文本我处理它并将键/值对传递给reducer我创建一个json ..所以有键但没有值所以mapper：

输入：文字/文字

输出：文本/文本

减速器：文本/文本

输出：文本/无

我的签名如下：

public class AdvanceCounter { /** * The map class of WordCount. */ public static class TokenCounterMapper extends Mapper { // <--- See this signature public void map(Object key, Text value, Context context) // <--- See this signature throws IOException, InterruptedException { context.write(key,value); //both are of type text OUTPUT TO REDUCER } } public static class TokenCounterReducer extends Reducer { // <--- See this signature Nullwritable here public void reduce(Text key, Iterable values, Context context) // <--- See this signature throws IOException, InterruptedException { for (Text value : values) { JSONObject jsn = new JSONObject(); //String output = ""; String[] vals = value.toString().split("\t"); String[] targetNodes = vals[0].toString().split(",",-1); try { jsn.put("source",vals[1]); jsn.put("targets",targetNodes); context.write(new Text(jsn.toString()),null); // no value } catch (JSONException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); Job job = new Job(conf, "Example Hadoop 0.20.1 WordCount"); // ... // job.setOutputValueClass(NullWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } 

但在执行时，我收到此错误：

 13/06/04 13:08:26 INFO mapred.JobClient: Task Id : attempt_201305241622_0053_m_000008_0, Status : FAILED java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.NullWritable, recieved org.apache.hadoop.io.Text at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1019) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691) at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) at org.sogou.Stinger$TokenCounterMapper.map(Stinger.java:72) at org.sogou.Stinger$TokenCounterMapper.map(Stinger.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093) at org.apache.hadoop.mapred.Child.main(Child.java:249) 

Answer 1

您尚未指定地图输出类型，因此它与您为reducer设置的相同，即Text和NullWritable ，它对于您的mapper不正确。 您应该执行以下操作以避免任何混淆，最好为mapper和reducer指定所有类型：

 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Text.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class);