1.RetriesExhaustedException
有两类:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy to /10.37.17.248:60020 after attempts=1
此类是由于映射ip地址时出错,需将/etc/hosts中的::1行去掉,改为:
127.0.0.1 localhost
第二类:
2017-07-11 18:59:19,283 INFO [MASTER_SERVER_OPERATIONS-hykj-master-3:60020-2] handler.ServerShutdownHandler: Received exception accessing hbase:meta during server shutdown of hykj-slave-5,60040,1499770355359, retrying hbase:meta read
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=351, exceptions:
Tue Jul 11 18:59:19 CST 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68208: row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hykj-slave-5,60040,1499770355359, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:264)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:219)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:56)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:299)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:278)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:845)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:600)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:156)
at org.apache.hadoop.hbase.MetaTableAccessor.getServerUserRegions(MetaTableAccessor.java:553)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:176)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68208: row '' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=hykj-slave-5,60040,1499770355359, seqNum=0
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:314)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:295)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
网上找到可能的原因是:
由于写入数据量大,进一步触发了HBase的Region Split操作,这一过程一般耗时较长,且Region Split操作触发较为频繁;
同时由于Region Split操作导致Region分布不均匀,进而触发HBase自动做Region Balance操作,Region迁移过程中也会导致Region下线,这一过程耗时较长。
本例是第一类问题。