我在测试hadoop时候,自己写了一个类,并且这个类中引用了2个类,运行的时候找不到引用的类。
StudentinfoRecord这个类确实有,并且和DBInput在同一目录下hduser@tianbaoxing-virtual-machine:/app/hadoop/tmp/java$ hadoop DBInputava$ hadoop DBInputWarning: $HADOOP_HOME is deprecated.13/04/24 10:41:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/04/24 10:41:59 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/04/24 10:42:00 INFO mapred.JobClient: Running job: job_201304241011_0004
13/04/24 10:42:01 INFO mapred.JobClient: map 0% reduce 0%
13/04/24 10:42:12 INFO mapred.JobClient: Task Id : attempt_201304241011_0004_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: StudentinfoRecordat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)at org.apache.hadoop.mapred.lib.db.DBConfiguration.getInputClass(DBConfiguration.java:192)at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:284)at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)at org.apache.hadoop.mapred.Child$4.run(Child.java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: StudentinfoRecordat java.net.URLClassLoader$1.run(URLClassLoader.java:202)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(URLClassLoader.java:190)at java.lang.ClassLoader.loadClass(ClassLoader.java:306)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)at java.lang.ClassLoader.loadClass(ClassLoader.java:247)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:247)at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)... 10 more13/04/24 10:42:12 INFO mapred.JobClient: Task Id : attempt_201304241011_0004_m_000001_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: StudentinfoRecordat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)at org.apache.hadoop.mapred.lib.db.DBConfiguration.getInputClass(DBConfiguration.java:192)at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:284)at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)at org.apache.hadoop.mapred.Child$4.run(Child.java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: StudentinfoRecordat java.net.URLClassLoader$1.run(URLClassLoader.java:202)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(URLClassLoader.java:190)at java.lang.ClassLoader.loadClass(ClassLoader.java:306)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)at java.lang.ClassLoader.loadClass(ClassLoader.java:247)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:247)at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)... 10 more13/04/24 10:42:21 INFO mapred.JobClient: Task Id : attempt_201304241011_0004_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: StudentinfoRecordat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)at org.apache.hadoop.mapred.lib.db.DBConfiguration.getInputClass(DBConfiguration.java:192)at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:284)at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)at org.apache.hadoop.mapred.Child$4.run(Child.java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: StudentinfoRecordat java.net.URLClassLoader$1.run(URLClassLoader.java:202)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(URLClassLoader.java:190)at java.lang.ClassLoader.loadClass(ClassLoader.java:306)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)at java.lang.ClassLoader.loadClass(ClassLoader.java:247)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:247)at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)... 10 more13/04/24 10:42:21 INFO mapred.JobClient: Task Id : attempt_201304241011_0004_m_000001_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: StudentinfoRecordat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)at org.apache.hadoop.mapred.lib.db.DBConfiguration.getInputClass(DBConfiguration.java:192)at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:284)at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)at org.apache.hadoop.mapred.Child$4.run(Child.java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: StudentinfoRecordat java.net.URLClassLoader$1.run(URLClassLoader.java:202)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(URLClassLoader.java:190)at java.lang.ClassLoader.loadClass(ClassLoader.java:306)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)at java.lang.ClassLoader.loadClass(ClassLoader.java:247)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:247)at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)... 10 more13/04/24 10:42:28 INFO mapred.JobClient: Task Id : attempt_201304241011_0004_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: StudentinfoRecordat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)at org.apache.hadoop.mapred.lib.db.DBConfiguration.getInputClass(DBConfiguration.java:192)at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:284)at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)at org.apache.hadoop.mapred.Child$4.run(Child.java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: StudentinfoRecordat java.net.URLClassLoader$1.run(URLClassLoader.java:202)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(URLClassLoader.java:190)at java.lang.ClassLoader.loadClass(ClassLoader.java:306)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)at java.lang.ClassLoader.loadClass(ClassLoader.java:247)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:247)at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)... 10 more13/04/24 10:42:29 INFO mapred.JobClient: Task Id : attempt_201304241011_0004_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: StudentinfoRecordat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:849)at org.apache.hadoop.mapred.lib.db.DBConfiguration.getInputClass(DBConfiguration.java:192)at org.apache.hadoop.mapred.lib.db.DBInputFormat.getRecordReader(DBInputFormat.java:284)at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)at org.apache.hadoop.mapred.Child$4.run(Child.java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException: StudentinfoRecordat java.net.URLClassLoader$1.run(URLClassLoader.java:202)at java.security.AccessController.doPrivileged(Native Method)at java.net.URLClassLoader.findClass(URLClassLoader.java:190)at java.lang.ClassLoader.loadClass(ClassLoader.java:306)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)at java.lang.ClassLoader.loadClass(ClassLoader.java:247)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:247)at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:847)... 10 more13/04/24 10:42:40 INFO mapred.JobClient: Job complete: job_201304241011_0004
13/04/24 10:42:40 INFO mapred.JobClient: Counters: 6
13/04/24 10:42:40 INFO mapred.JobClient: Job Counters
13/04/24 10:42:40 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=70989
13/04/24 10:42:40 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/04/24 10:42:40 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/04/24 10:42:40 INFO mapred.JobClient: Launched map tasks=8
13/04/24 10:42:40 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13/04/24 10:42:40 INFO mapred.JobClient: Failed map tasks=1
13/04/24 10:42:40 INFO mapred.JobClient: Job Failed: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201304241011_0004_m_000000
Exception in thread "main" java.io.IOException: Job failed!at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1327)at DBInput.main(DBInput.java:37)
hduser@tianbaoxing-virtual-machine:/app/hadoop/tmp/java$
这个问题找到了,我使用的是1.1.2,需要打包成jar文件运行。然后就没有报错。
程序不是以jar的形式运行的,所以不会上传jar到HDFS中,以致节点外的所有节点在执行task任务时上不能找到map和reduce类,所以在运行task时会出现错误。
同时还要注意命令: hadoop jar DBInput.jar ,中间不能省去jar
还要注意: DBInput.jar必须放在当前的文件下
即时是伪分布式下(一个节点既是主节点又是从节点)下直接在eclipse中也不能运行成功,也需要打包成jar