检测缺失块
1 hdfs fsck -list-corruptfileblocks
1 hdfs fsck / | egrep -v '^\.+$' | grep -v eplica
查看上面某一个文件的情况
1 hdfs fsck /path/to/corrupt/file -locations -blocks -files
解决方法
- 如果文件不重要，可以直接删除此文件；或删除后重新复制一份到集群中
- 如果不能删除，需要从上面命令中找到发生在哪台机器上，然后到此机器上查看日志。
参考
- http://centoshowtos.org/hadoop/fix-corrupt-blocks-on-hdfs/
- http://stackoverflow.com/questions/19205057/how-to-fix-corrupt-hadoop-hdfs
- HDFS block丢失过多进入安全模式（Safe mode）的解决方法
-
背景及现象描述(Background and Symptom)
因磁盘空间不足，内存不足，系统掉电等其他原因导致dataNode datablock丢失，出现如下类似日志：
The number of live datanodes 3 has reached the minimum number 0.
Safe mode will be turned off automatically once the thresholds have been reached.
Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled.
Name node is in safe mode.
The reported blocks 632758 needs additional 5114 blocks to reach the threshold 0.9990
of total blocks 638510.
The number of live datanodes 3 has reached the minimum number 0.
Safe mode will be turned off automatically once the thresholds have been reached.
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode
(FSNamesystem.java:1209)
... 12 more
原因分析(Cause Analysis)*
由于系统断电，内存不足等原因导致dataNode丢失超过设置的丢失百分比，系统自动进入安全模式
解决办法(Solution)*
安装HDFS客户端，并执行如下命令：
步骤 1 执行命令退出安全模式：hadoop dfsadmin -safemode leave
步骤 2 执行健康检查，删除损坏掉的block。 hdfs fsck / -delete
注意: 这种方式会出现数据丢失，损坏的block会被删掉
