hadoop wordcount异常，hadoopwordcount
hadoop wordcount异常，hadoopwordcount
最近学习hadoop，在windows+eclipse+虚拟机hadoop集群环境下运行mapreduce程序遇到了很多问题。上网查了查，并经过自己的分析，最终解决，在此分享一下，给遇到同样问题的人提供参考。
我的hadoop集群环境：
虚拟机上4台机器：192.168.137.111（master）、192.168.137.112（slave1）、192.168.137.113（slave2）、192.168.137.114（slave3）
hadoop集群用户名：hadoop
hadoop版本：hadoop-1.1.2
开发环境：windows7+eclipse+hadoop插件
异常1：
14/10/18 08:23:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/10/18 08:23:47 ERROR security.UserGroupInformation: PriviledgedActionException as:guilin cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-guilin\mapred\staging\guilin1651756173\.staging to 0700 Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-guilin\mapred\staging\guilin1651756173\.staging to 0700 at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:689) at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:662) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:918) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912) at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) at com.guilin.hadoop.mapreduce.WordCount.main(WordCount.java:75)原因：wordcount程序连的是本地windows上的hadoop，需添加conf.set("mapred.job.tracker", "master:9001")，连接集群。
异常2：
14/10/18 08:37:14 ERROR security.UserGroupInformation: PriviledgedActionException as:guilin cause:org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=guilin, access=EXECUTE, inode="hadoop":hadoop:supergroup:rwx------ Exception in thread "main" org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=guilin, access=EXECUTE, inode="hadoop":hadoop:supergroup:rwx------ at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1030) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:918) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912) at org.apache.hadoop.mapreduce.Job.submit(Job.java:500) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530) at com.guilin.hadoop.mapreduce.WordCount.main(WordCount.java:75) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Permission denied: user=guilin, access=EXECUTE, inode="hadoop":hadoop:supergroup:rwx------ at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:199) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:155) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:125) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5468) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkTraverse(FSNamesystem.java:5447) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:2168) at org.apache.hadoop.hdfs.server.namenode.NameNode.getFileInfo(NameNode.java:888) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:230) at com.sun.proxy.$Proxy2.getFileInfo(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy2.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1028) ... 12 more原因：wordcount程序使用windows7的账户登录集群hadoop，我的系统账户名是guilin，而hadoop集群账户是hadoop，并且集群hadoop目录权限设置的是仅hadoop用户有读、写、执行权限。
解决办法：第一种是修改windows管理员（Administrator）账户名为hadoop账户名；第二种是在集群上创建一个账户名称与windows管理员账户名相同，并设置对hadoop目录有读、写、执行权限。推荐使用第一种，管理windows用户详情见http://jingyan.baidu.com/article/02027811ac3dc21bcc9ce59a.html 。
异常3：
14/10/18 09:57:19 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 14/10/18 09:57:19 INFO input.FileInputFormat: Total input paths to process : 5 14/10/18 09:57:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/10/18 09:57:19 WARN snappy.LoadSnappy: Snappy native library not loaded 14/10/18 09:57:20 INFO mapred.JobClient: Running job: job_201410181754_0001 14/10/18 09:57:21 INFO mapred.JobClient: map 0% reduce 0% 14/10/18 09:57:29 INFO mapred.JobClient: Task Id : attempt_201410181754_0001_m_000004_0, Status : FAILED java.lang.RuntimeException: java.lang.ClassNotFoundException: com.guilin.hadoop.mapreduce.WordCount$TokenizerMapper at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849) at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:199) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:719) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249)
原因：hadoop集群上运行mapreduce程序需要jar包。
解决办法：添加conf.set("mapred.jar","hadoop-test.jar");
把项目打包为jar文件hadoop-test.jar，放置在项目根目录下。
wordcount完整代码
package com.guilin.hadoop.mapreduce;
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class WordCount {
public static class TokenizerMapper extends
Mapper<Object, Text, Text, IntWritable> {
private static final IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value,
Mapper<Object, Text, Text, IntWritable>.Context context)
throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
this.word.set(itr.nextToken());
context.write(this.word, one);
}
}
}
public static class IntSumReducer extends
Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Reducer<Text, IntWritable, Text, IntWritable>.Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
this.result.set(sum);
context.write(key, this.result);
}
}
public static void main(String[] args) throws IOException,
ClassNotFoundException, InterruptedException {
Configuration conf = new Configuration();
conf.set("mapred.job.tracker", "master:9001");
conf.set("mapred.jar", "hadoop-test.jar");
String[] ars = new String[] { "hdfs://master:9000/usr/hadoop/input",
"hdfs://master:9000/usr/hadoop/newout1" };
String[] otherArgs = new GenericOptionsParser(conf, ars)
.getRemainingArgs();
if (otherArgs.length != 2) {
System.err.println("Usage: wordcount <in> <out>");
System.exit(2);
}
Job job = new Job(conf, "wordcount");
job.setJarByClass(WordCount.class);
job.setMapperClass(WordCount.TokenizerMapper.class);
job.setCombinerClass(WordCount.IntSumReducer.class);
job.setReducerClass(WordCount.IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
最后运行成功
14/10/18 10:12:27 INFO input.FileInputFormat: Total input paths to process : 2 14/10/18 10:12:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/10/18 10:12:27 WARN snappy.LoadSnappy: Snappy native library not loaded 14/10/18 10:12:27 INFO mapred.JobClient: Running job: job_201410181754_0004 14/10/18 10:12:28 INFO mapred.JobClient: map 0% reduce 0% 14/10/18 10:12:32 INFO mapred.JobClient: map 100% reduce 0% 14/10/18 10:12:39 INFO mapred.JobClient: map 100% reduce 33% 14/10/18 10:12:40 INFO mapred.JobClient: map 100% reduce 100% 14/10/18 10:12:40 INFO mapred.JobClient: Job complete: job_201410181754_0004 14/10/18 10:12:40 INFO mapred.JobClient: Counters: 29 14/10/18 10:12:40 INFO mapred.JobClient: Job Counters 14/10/18 10:12:40 INFO mapred.JobClient: Launched reduce tasks=1 14/10/18 10:12:40 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=4614 14/10/18 10:12:40 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 14/10/18 10:12:40 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 14/10/18 10:12:40 INFO mapred.JobClient: Launched map tasks=2 14/10/18 10:12:40 INFO mapred.JobClient: Data-local map tasks=2 14/10/18 10:12:40 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=8329 14/10/18 10:12:40 INFO mapred.JobClient: File Output Format Counters 14/10/18 10:12:40 INFO mapred.JobClient: Bytes Written=31 14/10/18 10:12:40 INFO mapred.JobClient: FileSystemCounters 14/10/18 10:12:40 INFO mapred.JobClient: FILE_BYTES_READ=75 14/10/18 10:12:40 INFO mapred.JobClient: HDFS_BYTES_READ=264 14/10/18 10:12:40 INFO mapred.JobClient: FILE_BYTES_WRITTEN=154204 14/10/18 10:12:40 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=31 14/10/18 10:12:40 INFO mapred.JobClient: File Input Format Counters 14/10/18 10:12:40 INFO mapred.JobClient: Bytes Read=44 14/10/18 10:12:40 INFO mapred.JobClient: Map-Reduce Framework 14/10/18 10:12:40 INFO mapred.JobClient: Map output materialized bytes=81 14/10/18 10:12:40 INFO mapred.JobClient: Map input records=2 14/10/18 10:12:40 INFO mapred.JobClient: Reduce shuffle bytes=81 14/10/18 10:12:40 INFO mapred.JobClient: Spilled Records=12 14/10/18 10:12:40 INFO mapred.JobClient: Map output bytes=78 14/10/18 10:12:40 INFO mapred.JobClient: CPU time spent (ms)=1090 14/10/18 10:12:40 INFO mapred.JobClient: Total committed heap usage (bytes)=241246208 14/10/18 10:12:40 INFO mapred.JobClient: Combine input records=8 14/10/18 10:12:40 INFO mapred.JobClient: SPLIT_RAW_BYTES=220 14/10/18 10:12:40 INFO mapred.JobClient: Reduce input records=6 14/10/18 10:12:40 INFO mapred.JobClient: Reduce input groups=4 14/10/18 10:12:40 INFO mapred.JobClient: Combine output records=6 14/10/18 10:12:40 INFO mapred.JobClient: Physical memory (bytes) snapshot=311574528 14/10/18 10:12:40 INFO mapred.JobClient: Reduce output records=4 14/10/18 10:12:40 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1034760192 14/10/18 10:12:40 INFO mapred.JobClient: Map output records=8
Unable to load native-hadoop libra ry for your platform... using builtin-java classes where applicable
这两个是警告没事 你到底是啥问题呢？出去这两个还有其他问题吗？
只能查看hadoop的日志来解决了，在hadoop的安装路径中有个logs的文件夹，试着查查看吧
评论暂时关闭