设为首页 加入收藏

TOP

HBase配置HMaster死亡
2019-04-29 01:48:06 】 浏览:40
Tags:HBase 配置 HMaster 死亡

在配置HBase的时候,我们经常遇到一个问题就是regionserver正常运行,但是HMaster进程会在开始启动之后的十几秒内退出。使得HBase配置失效,经常发生的错误如下所示:

2015-06-06 17:23:33,232 INFO [master1:16020.activeMasterManager] master.SplitLogManager: dead splitlog workers [slave1,16020,1433581428769]
2015-06-06 17:23:33,234 INFO [master1:16020.activeMasterManager] master.SplitLogManager: hdfs://master1:9000/hbase/WALs/slave1,16020,1433581428769-splitting is empty dir, no logs to split
2015-06-06 17:23:33,258 INFO [master1:16020.activeMasterManager] master.SplitLogManager: started splitting 0 logs in [hdfs://master1:9000/hbase/WALs/slave1,16020,1433581428769-splitting] for [slave1,16020,1433581428769]
2015-06-06 17:23:33,265 INFO [master1:16020.activeMasterManager] master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://master1:9000/hbase/WALs/slave1,16020,1433581428769-splitting] in 7ms
2015-06-06 17:23:33,265 INFO [master1:16020.activeMasterManager] zookeeper.MetaTableLocator: Deleting hbase:meta region location in ZooKeeper
2015-06-06 17:23:33,306 INFO [master1:16020.activeMasterManager] master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to slave2,16020,1433582611910
2015-06-06 17:23:33,306 INFO [master1:16020.activeMasterManager] master.RegionStates: Transition {1588230740 state=OFFLINE, ts=1433582613274, server=null} to {1588230740 state=PENDING_OPEN, ts=1433582613306, server=slave2,16020,1433582611910}
2015-06-06 17:23:33,423 INFO [master1:16020.activeMasterManager] master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
2015-06-06 17:23:33,477 INFO [AM.ZK.Worker-pool2-t1] master.RegionStates: Transition {1588230740 state=PENDING_OPEN, ts=1433582613306, server=slave2,16020,1433582611910} to {1588230740 state=OPENING, ts=1433582613477, server=slave2,16020,1433582611910}
2015-06-06 17:23:33,891 INFO [AM.ZK.Worker-pool2-t2] master.RegionStates: Transition {1588230740 state=OPENING, ts=1433582613477, server=slave2,16020,1433582611910} to {1588230740 state=OPEN, ts=1433582613891, server=slave2,16020,1433582611910}
2015-06-06 17:23:33,893 INFO [AM.ZK.Worker-pool2-t2] coordination.ZkOpenRegionCoordination: Handling OPENED of 1588230740 from master1,16020,1433581867821; deleting unassigned node
2015-06-06 17:23:33,898 INFO [AM.ZK.Worker-pool2-t3] master.RegionStates: Onlined 1588230740 on slave2,16020,1433582611910
2015-06-06 17:23:33,904 INFO [master1:16020.activeMasterManager] master.HMaster: hbase:meta assigned=1, rit=false, location=slave2,16020,1433582611910
2015-06-06 17:23:34,027 INFO [master1:16020.activeMasterManager] hbase.MetaMigrationConvertingToPB: hbase:meta doesn't have any entries to update.
2015-06-06 17:23:34,027 INFO [master1:16020.activeMasterManager] hbase.MetaMigrationConvertingToPB: META already up-to date with PB serialization
2015-06-06 17:23:34,037 INFO [master1:16020.activeMasterManager] master.AssignmentManager: Clean cluster startup. Assigning user regions
2015-06-06 17:23:34,039 INFO [master1:16020.activeMasterManager] master.AssignmentManager: Joined the cluster in 11ms, failover=false
2015-06-06 17:23:34,079 INFO [master1:16020.activeMasterManager] master.TableNamespaceManager: Namespace table not found. Creating...
2015-06-06 17:23:34,115 FATAL [master1:16020.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.checkAndSetEnablingTable(CreateTableHandler.java:151)
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:124)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:233)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:871)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:722)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:165)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1428)
at java.lang.Thread.run(Thread.java:745)
2015-06-06 17:23:34,116 FATAL [master1:16020.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: []
2015-06-06 17:23:34,116 FATAL [master1:16020.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.checkAndSetEnablingTable(CreateTableHandler.java:151)
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:124)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:233)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:871)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:722)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:165)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1428)
at java.lang.Thread.run(Thread.java:745)
2015-06-06 17:23:34,117 INFO [master1:16020.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
2015-06-06 17:23:34,117 INFO [master/master1/192.168.1.107:16020] regionserver.HRegionServer: Stopping infoServer
2015-06-06 17:23:34,151 INFO [master/master1/192.168.1.107:16020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010
2015-06-06 17:23:34,253 INFO [master/master1/192.168.1.107:16020] regionserver.HRegionServer: stopping server master1,16020,1433581867821
2015-06-06 17:23:34,263 INFO [master/master1/192.168.1.107:16020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x24dc68e699e0007
2015-06-06 17:23:34,271 INFO [master/master1/192.168.1.107:16020] zookeeper.ZooKeeper: Session: 0x24dc68e699e0007 closed
2015-06-06 17:23:34,272 INFO [master/master1/192.168.1.107:16020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-06 17:23:34,277 INFO [master/master1/192.168.1.107:16020] regionserver.HRegionServer: stopping server master1,16020,1433581867821; all regions closed.
2015-06-06 17:23:34,280 INFO [master1:16020.archivedHFileCleaner] cleaner.HFileCleaner: master1:16020.archivedHFileCleaner exiting
2015-06-06 17:23:34,280 INFO [master1,16020,1433581867821-BalancerChore] balancer.BalancerChore: master1,16020,1433581867821-BalancerChore exiting
2015-06-06 17:23:34,280 INFO [master1:16020.oldLogCleaner] cleaner.LogCleaner: master1:16020.oldLogCleaner exiting
2015-06-06 17:23:34,280 INFO [master1:16020.oldLogCleaner] master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x24dc68e699e0008, quorum=slave1:2181,master1:2181,slave2:2181, baseZNode=/hbase
2015-06-06 17:23:34,281 INFO [CatalogJanitor-master1:16020] master.CatalogJanitor: CatalogJanitor-master1:16020 exiting
2015-06-06 17:23:34,281 INFO [master1,16020,1433581867821-ClusterStatusChore] balancer.ClusterStatusChore: master1,16020,1433581867821-ClusterStatusChore exiting
2015-06-06 17:23:34,283 INFO [master1:16020.oldLogCleaner] zookeeper.ZooKeeper: Session: 0x24dc68e699e0008 closed
2015-06-06 17:23:34,283 INFO [master1:16020.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-06 17:23:34,286 INFO [master/master1/192.168.1.107:16020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x34dc7d3c9ed0002
2015-06-06 17:23:34,289 INFO [master1:16020.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-06 17:23:34,290 INFO [master/master1/192.168.1.107:16020] zookeeper.ZooKeeper: Session: 0x34dc7d3c9ed0002 closed
2015-06-06 17:23:34,391 INFO [master1,16020,1433581867821.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor: master1,16020,1433581867821.splitLogManagerTimeoutMonitor exiting
2015-06-06 17:23:34,391 INFO [master/master1/192.168.1.107:16020] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2015-06-06 17:23:34,391 INFO [master/master1/192.168.1.107:16020] ipc.RpcServer: Stopping server on 16020
2015-06-06 17:23:34,392 INFO [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: stopping
2015-06-06 17:23:34,394 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2015-06-06 17:23:34,394 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2015-06-06 17:23:34,404 INFO [master/master1/192.168.1.107:16020] zookeeper.RecoverableZooKeeper: Node /hbase/rs/master1,16020,1433581867821 already deleted, retry=false
2015-06-06 17:23:34,409 INFO [master/master1/192.168.1.107:16020] zookeeper.ZooKeeper: Session: 0x34dc7d3c9ed0001 closed
2015-06-06 17:23:34,409 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2015-06-06 17:23:34,409 INFO [master/master1/192.168.1.107:16020] regionserver.HRegionServer: stopping server master1,16020,1433581867821; zookeeper connection closed.
2015-06-06 17:23:34,410 INFO [master/master1/192.168.1.107:16020] regionserver.HRegionServer: master/master1/192.168.1.107:16020 exiting

发生这样错误的主要原因是因为zookeeper与hbase连接出现问题,hbase需要zookeeper来协调管理元数据的存储以及数据的管理。解决方法是将两者冲突的文件删除掉,这些文件一般我们自己配置的文件。重新启动集群环境。


】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇LSM树原理及应用到HBase的索引 下一篇非关系型数据库Hbase与Redis:

最新文章

热门文章

Hot 文章

Python

C 语言

C++基础

大数据基础

linux编程基础

C/C++面试题目