设为首页 加入收藏

TOP

hbase 删除部门wal日志后自我修复
2019-02-09 01:50:18 】 浏览:131
Tags:hbase 删除 部门 wal 日志 自我 修复

wal日志被部分删除后,正常启动hbase,hbase先尝试读数据,不断报错,经过大约20分的挣扎后最终发现日志丢失,程序放弃读wal日志,正常启动hbase。wal日志删除的时候没有写数据,数据没有丢失,若刚写完wal日志,还没写数据文件,那估计数据会少量丢失,但是程序会提示写入失败;总体来说hbase的容错能力还是棒棒的大笑


详细错误日志如下:

1、刚开始报错的日志

2013-12-20 23:39:32,786 WARN [SplitLogWorker-hadoop03,60020,1387553065927] hdfs.DFSClient: Last block locations unavailable. Datanodes might not have reported blocks completely. Will retry for 2 times
2013-12-20 23:39:36,797 WARN [SplitLogWorker-hadoop03,60020,1387553065927] hdfs.DFSClient: Last block locations unavailable. Datanodes might not have reported blocks completely. Will retry for 1 times
2013-12-20 23:39:40,798 WARN [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogFactory: Lease should have recovered. This is not expected. Will retry
java.io.IOException: Could not obtain the last block locations.
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1958)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1936)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:731)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:165)
        at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1499)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:76)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:69)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.reset(SequenceFileLogReader.java:174)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.initReader(SequenceFileLogReader.java:183)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:68)
        at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:113)
        at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:645)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:554)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:273)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:217)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:138)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:358)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:245)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:205)
        at java.lang.Thread.run(Thread.java:662)
2013-12-20 23:39:41,307 WARN [SplitLogWorker-hadoop03,60020,1387553065927] hdfs.DFSClient: Last block locations unavailable. Datanodes might not have reported blocks completely. Will retry for 3 times
2013-12-20 23:39:45,310 WARN [SplitLogWorker-hadoop03,60020,1387553065927] hdfs.DFSClient: Last block locations unavailable. Datanodes might not have reported blocks completely. Will retry for 2 times
2013-12-20 23:39:49,313 WARN [SplitLogWorker-hadoop03,60020,1387553065927] hdfs.DFSClient: Last block locations unavailable. Datanodes might not have reported blocks completely. Will retry for 1 times
2013-12-20 23:39:53,821 WARN [SplitLogWorker-hadoop03,60020,1387553065927] hdfs.DFSClient: Last block locations unavailable. Datanodes might not have reported blocks completely. Will retry for 3 times

2、发现日志丢失,浴火重生后的日志

2013-12-20 23:44:31,032 WARN [SplitLogWorker-hadoop03,60020,1387553065927] hdfs.DFSClient: Last block locations unavailable. Datanodes might not have reported blocks completely. Will retry for 2 times
2013-12-20 23:44:35,034 WARN [SplitLogWorker-hadoop03,60020,1387553065927] hdfs.DFSClient: Last block locations unavailable. Datanodes might not have reported blocks completely. Will retry for 1 times
2013-12-20 23:44:39,035 ERROR [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogFactory: Can't open after 24 attempts and 310267ms for hdfs://hadoop01:9000/hbase/WALs/hadoop04,60020,1387532758006-splitting/hadoop04%2C60020%2C1387532758006.1387547628540.meta
2013-12-20 23:44:39,035 WARN [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogSplitter: Could not get reader, corrupted log file hdfs://hadoop01:9000/hbase/WALs/hadoop04,60020,1387532758006-splitting/hadoop04%2C60020%2C1387532758006.1387547628540.meta
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter$CorruptedLogFileException: skipErrors=true Could not open hlog hdfs://hadoop01:9000/hbase/WALs/hadoop04,60020,1387532758006-splitting/hadoop04%2C60020%2C1387532758006.1387547628540.meta ignoring
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:578)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:273)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:217)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:138)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:358)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:245)
        at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:205)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Could not obtain the last block locations.
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1958)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1936)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:731)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:165)
        at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1499)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:76)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:69)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.reset(SequenceFileLogReader.java:174)
        at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.initReader(SequenceFileLogReader.java:183)
        at org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:68)
        at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:113)
        at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:645)
        at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:554)
        ... 7 more
2013-12-20 23:44:39,049 WARN [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogSplitter: Nothing to split in log file hdfs://hadoop01:9000/hbase/WALs/hadoop04,60020,1387532758006-splitting/hadoop04%2C60020%2C1387532758006.1387547628540.meta
2013-12-20 23:44:39,049 INFO [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogSplitter: Finishing writing output logs and closing down.
2013-12-20 23:44:39,049 INFO [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogSplitter: Processed 0 edits across 0 regions; log file=hdfs://hadoop01:9000/hbase/WALs/hadoop04,60020,1387532758006-splitting/hadoop04%2C60020%2C1387532758006.1387547628540.meta is corrupted = true progress failed = false
2013-12-20 23:44:39,056 INFO [SplitLogWorker-hadoop03,60020,1387553065927] regionserver.SplitLogWorker: successfully transitioned task /hbase/splitWAL/WALs%2Fhadoop04%2C60020%2C1387532758006-splitting%2Fhadoop04%252C60020%252C1387532758006.1387547628540.meta to final state DONE hadoop03,60020,1387553065927
2013-12-20 23:44:39,056 INFO [SplitLogWorker-hadoop03,60020,1387553065927] regionserver.SplitLogWorker: worker hadoop03,60020,1387553065927 done with task /hbase/splitWAL/WALs%2Fhadoop04%2C60020%2C1387532758006-splitting%2Fhadoop04%252C60020%252C1387532758006.1387547628540.meta in 1210589ms
2013-12-20 23:44:39,079 DEBUG [regionserver60020-EventThread] regionserver.SplitLogWorker: tasks arrived or departed
2013-12-20 23:44:44,454 DEBUG [regionserver60020-EventThread] regionserver.SplitLogWorker: tasks arrived or departed
2013-12-20 23:44:44,461 INFO [SplitLogWorker-hadoop03,60020,1387553065927] regionserver.SplitLogWorker: worker hadoop03,60020,1387553065927 acquired task /hbase/splitWAL/WALs%2Fhadoop03%2C60020%2C1387532758114-splitting%2Fhadoop03%252C60020%252C1387532758114.1387546149859
2013-12-20 23:44:44,464 INFO [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogSplitter: Splitting hlog: hdfs://hadoop01:9000/hbase/WALs/hadoop03,60020,1387532758114-splitting/hadoop03%2C60020%2C1387532758114.1387546149859, length=0
2013-12-20 23:44:44,464 INFO [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogSplitter: DistributedLogReplay = false
2013-12-20 23:44:44,469 WARN [SplitLogWorker-hadoop03,60020,1387553065927] wal.HLogSplitter: File hdfs://hadoop01:9000/hbase/WALs/hadoop03,60020,1387532758114-splitting/hadoop03%2C60020%2C1387532758114.1387546149859 might be still open, length is 0
2013-12-20 23:44:44,469 INFO [SplitLogWorker-hadoop03,60020,1387553065927] util.FSHDFSUtils: Recovering lease on dfs file hdfs://hadoop01:9000/hbase/WALs/hadoop03,60020,1387532758114-splitting/hadoop03%2C60020%2C1387532758114.1387546149859
2013-12-20 23:44:44,471 INFO [SplitLogWorker-hadoop03,60020,1387553065927] util.FSHDFSUtils: recoverLease=false, attempt=0 on file=hdfs://hadoop01:9000/hbase/WALs/hadoop03,60020,1387532758114-splitting/hadoop03%2C60020%2C1387532758114.1387546149859 after 2ms
2013-12-20 23:44:44,965 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_06,180,1386570352699.eff5b3d3f1d657af7f0098cdf2beac24.
2013-12-20 23:44:45,431 DEBUG [regionserver60020-EventThread] regionserver.SplitLogWorker: tasks arrived or departed
2013-12-20 23:44:45,738 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_06,350,1386570352700.93d4f40723f0340eda17bc1022692442.
2013-12-20 23:44:45,739 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_06,600,1386570352701.0a5c24fdb316dc1b1538ce96d352484f.
2013-12-20 23:44:45,740 DEBUG [RS_OPEN_REGION-hadoop03:60020-1] zookeeper.ZKAssign: regionserver:60020-0x1431091a836000b Transitioning 93d4f40723f0340eda17bc1022692442 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2013-12-20 23:44:45,740 DEBUG [RS_OPEN_REGION-hadoop03:60020-0] zookeeper.ZKAssign: regionserver:60020-0x1431091a836000b Transitioning eff5b3d3f1d657af7f0098cdf2beac24 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2013-12-20 23:44:45,742 DEBUG [RS_OPEN_REGION-hadoop03:60020-2] zookeeper.ZKAssign: regionserver:60020-0x1431091a836000b Transitioning 0a5c24fdb316dc1b1538ce96d352484f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2013-12-20 23:44:45,742 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_06,905,1386570352703.f05316cc10da1ba64f69b3d3536bc722.
2013-12-20 23:44:45,742 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_07,180,1386570372569.f84250e5877006bd2212da2da0cb3cd3.
2013-12-20 23:44:45,749 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_07,395,1386570372571.d3a1f84157f0ecd7a1861af4a0b6b849.
2013-12-20 23:44:45,749 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_07,695,1386570372575.f5bae4f94fbc3bc7cac5aa636e6bcb33.
2013-12-20 23:44:45,750 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_07,875,1386570372577.73b614e4efe1dd502ad44e1a26d16987.
2013-12-20 23:44:45,750 INFO [Priority.RpcServer.handler=0,port=60020] regionserver.HRegionServer: Open call_1x_08,110,1387445594702.30678befee6063d4e5b2e4421baac6db.



有任何问题欢迎留言交流微笑


】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇spark从hbase读取写入数据 下一篇scala spark hbase 操作案例

最新文章

热门文章

Hot 文章

Python

C 语言

C++基础

大数据基础

linux编程基础

C/C++面试题目