hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar wordcount /user/chenxun/input/file.txt output
运行过程如下
17/10/1401:55:26 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:803217/10/1401:55:28 INFO input.FileInputFormat: Total input filestoprocess : 117/10/1401:55:28 INFO mapreduce.JobSubmitter: numberof splits:117/10/1401:55:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1507912953622_0017
17/10/1401:55:28 INFO impl.YarnClientImpl: Submitted application application_1507912953622_0017
17/10/1401:55:28 INFO mapreduce.Job: The url to track the job: http://chen.local:8088/proxy/application_1507912953622_0017/
17/10/1401:55:28 INFO mapreduce.Job: Running job: job_1507912953622_0017
17/10/1401:55:36 INFO mapreduce.Job: Job job_1507912953622_0017 running in uber mode : false17/10/1401:55:36 INFO mapreduce.Job: map 0% reduce 0%
17/10/1401:55:41 INFO mapreduce.Job: map 100% reduce 0%
17/10/1401:55:47 INFO mapreduce.Job: map 100% reduce 100%
17/10/1401:55:47 INFO mapreduce.Job: Job job_1507912953622_0017 completed successfully
17/10/1401:55:48 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number ofbytesread=44
FILE: Number ofbytes written=276523
FILE: Number ofread operations=0
FILE: Number of large read operations=0
FILE: Number ofwrite operations=0
HDFS: Number ofbytesread=141
HDFS: Number ofbytes written=26
HDFS: Number ofread operations=6
HDFS: Number of large read operations=0
HDFS: Number ofwrite operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3032
Total time spent by all reduces in occupied slots (ms)=3133
Total time spent by all map tasks (ms)=3032
Total time spent by all reduce tasks (ms)=3133
Total vcore-milliseconds taken by all map tasks=3032
Total vcore-milliseconds taken by all reduce tasks=3133
Total megabyte-milliseconds taken by all map tasks=3104768
Total megabyte-milliseconds taken by all reduce tasks=3208192
Map-Reduce Framework
Map input records=4
Map output records=4
Map output bytes=43
Map output materialized bytes=44
Input splitbytes=114
Combine input records=4
Combine output records=3
Reduce input groups=3
Reduce shuffle bytes=44
Reduce input records=3
Reduce output records=3
Spilled Records=6
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=134
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=306708480
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=27
File Output Format Counters
Bytes Written=26
今天在Mac配置伪分布式环境。部署完毕后,运行MapReduce程序,程序运行刚开始没有问题, 但是到启动Map Task时,就报错误,报exitCode: 127错误。
错误日志如下:
15/04/0600:08:01 INFO mapreduce.Job: Job job_1428250045856_0002 failed with state FAILED due to: Application application_1428250045856_0002 failed 2 times due to AM Container for appattempt_1428250045856_0002_000002 exited with exitCode: 127 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
解决办法:
1 从错误日志上看应该是某个Shell命令在Mac上没有找到。
2 查看每个程序的日志:hadoop-2.3.0/logs/userlogs/application_1428247759749_0002/container_1428247759749_0002_02_000001中的错误日志:
/bin/bash: /bin/java: No such file or directory
可以看到:/bin/java 的命令没有找到。
建立一个软链接,链接到java程序即可。
上文中的日志文件和异常中的文件名对应不上,这个是我编写时的问题, 实际可以根据日志中的文件名找到具体的错误文件。但是整个流程,是这个。
$ sudo ln -s /usr/bin/java /bin/java
再输入命令$ /bin/java -version验证一遍,显示:
引用
Java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java Hotspot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
这个时候你在执行sudo ln -s /usr/bin/java /bin/java出现问题:ln: /bin/java: Operation not permitted
这是mac系统的原因不是你的错。
这是因为苹果在OS X 10.11中引入的SIP特性使得即使加了sudo(也就是具有root权限)也无法修改系统级的目录,其中就包括了/usr/bin。要解决这个问题有两种做法:一种是比较不安全的就是关闭SIP,也就是rootless特性;另一种是将本要链接到/usr/bin下的改链接到/usr/local/bin下就好了。
废话不多说上解决方法: