你好,游客 登录 注册 搜索
背景:
阅读新闻

Hadoop异常 java.io.IOException: Job status not available

[日期:2016-11-23] 来源:Linux社区  作者:Linux [字体: ]

Hadoop集群上跑mapreduce,在job任务执行完成退出时报 java.io.IOException: Job status not available异常。Job client请求job状态时,Application已经完成转而去Job history server请求job状态,就在这里抛出异常。

[linuxidc@master conf]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /user/lizeyi/people.txt  /user/lizeyi/wordcount7
15/06/08 18:36:16 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.3.4.35:8032
15/06/08 18:36:17 INFO input.FileInputFormat: Total input paths to process : 1
15/06/08 18:36:17 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
15/06/08 18:36:17 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 39cf0c71a251a79c50555810ca660450d9682140]
15/06/08 18:36:17 INFO mapreduce.JobSubmitter: number of splits:1
15/06/08 18:36:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1433756996622_0004
15/06/08 18:36:18 INFO impl.YarnClientImpl: Submitted application application_1433756996622_0004
15/06/08 18:36:18 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1433756996622_0004/
15/06/08 18:36:18 INFO mapreduce.Job: Running job: job_1433756996622_0004
15/06/08 18:36:40 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
java.io.IOException: Job status not available 
        at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)
        at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:609)
        at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1354)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1316)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

须配置Job History Server相关参数,让Job Client可以读取job最后的执行状态,测试Hadoop版本2.5.0
添加参数vim mapred-site.xml
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>master.hadoop:10020</value>
  </property>
  <property>
    <name>yarn.app.mapreduce.am.staging-dir</name>
    <value>/tmp/hadoop-yarn/staging</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.intermediate-done-dir</name>
    <value>${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.done-dir</name>
    <value>${yarn.app.mapreduce.am.staging-dir}/history/done</value>
  </property>

修改完成配置后,重新执行任务后正常退出

[linuxidc@master conf]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /user/lizeyi/people.txt  /user/lizeyi/wordcount9
15/06/08 18:54:04 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.3.4.35:8032
15/06/08 18:54:06 INFO input.FileInputFormat: Total input paths to process : 1
15/06/08 18:54:06 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
15/06/08 18:54:06 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 39cf0c71a251a79c50555810ca660450d9682140]
15/06/08 18:54:06 INFO mapreduce.JobSubmitter: number of splits:1
15/06/08 18:54:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1433760669916_0001
15/06/08 18:54:07 INFO impl.YarnClientImpl: Submitted application application_1433760669916_0001
15/06/08 18:54:07 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1433760669916_0001/
15/06/08 18:54:07 INFO mapreduce.Job: Running job: job_1433760669916_0001
15/06/08 18:54:34 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
15/06/08 18:54:35 INFO mapreduce.Job: Job job_1433760669916_0001 running in uber mode : false
15/06/08 18:54:35 INFO mapreduce.Job:  map 100% reduce 100%
15/06/08 18:54:35 INFO mapreduce.Job: Job job_1433760669916_0001 completed successfully
15/06/08 18:54:35 INFO mapreduce.Job: Counters: 49
        File System Counters
                FILE: Number of bytes read=68
                FILE: Number of bytes written=205241
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=151
                HDFS: Number of bytes written=46
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=6036
                Total time spent by all reduces in occupied slots (ms)=6132
                Total time spent by all map tasks (ms)=6036
                Total time spent by all reduce tasks (ms)=6132
                Total vcore-seconds taken by all map tasks=6036
                Total vcore-seconds taken by all reduce tasks=6132
                Total megabyte-seconds taken by all map tasks=6180864
                Total megabyte-seconds taken by all reduce tasks=6279168
        Map-Reduce Framework
                Map input records=4
                Map output records=4
                Map output bytes=54
                Map output materialized bytes=68
                Input split bytes=113
                Combine input records=4
                Combine output records=4
                Reduce input groups=4
                Reduce shuffle bytes=68
                Reduce input records=4
                Reduce output records=4
                Spilled Records=8
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=82
                CPU time spent (ms)=2150
                Physical memory (bytes) snapshot=447897600
                Virtual memory (bytes) snapshot=1986359296
                Total committed heap usage (bytes)=355467264
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=38
        File Output Format Counters 
                Bytes Written=46

更多Hadoop相关信息见Hadoop 专题页面 http://www.linuxidc.com/topicnews.aspx?tid=13

 本文永久更新链接地址http://www.linuxidc.com/Linux/2016-11/137463.htm

linux
相关资讯       Hadoop异常 
本文评论   查看全部评论 (0)
表情: 表情 姓名: 字数

       

评论声明
  • 尊重网上道德,遵守中华人民共和国的各项有关法律法规
  • 承担一切因您的行为而直接或间接导致的民事或刑事法律责任
  • 本站管理人员有权保留或删除其管辖留言中的任意内容
  • 本站有权在网站内转载或引用您的评论
  • 参与本评论即表明您已经阅读并接受上述条款