设为首页 加入收藏

TOP

hbase 源代码分析 (9) hbase启动过程
2018-12-17 01:47:14 】 浏览:83
Tags:hbase 源代码 分析 启动 过程

上一章节:hbase 源代码分析 (8) delete 过程 详解

http://blog.csdn.net/chenfenggang/article/details/75094362


这一章节主要分析shell脚本。hbase的启动过程。

过程如下:

1)运行start-hbase.sh
2) 加载conf,加载需要lib,class文件,包括jdk里面的,hbase本身的。
3)判断安装模式
4)如果集群模式需要启动
a)zookeeper,
b)Master
c)RegionService
d)master-backup
如果本地模式则只需要启动master就够了。
在master里面会new一个zk,和启动一个regionService,但是这个master和regionSerivce是同一个JVM

启动maseter主要是调用org.apache.hadoop.hbase.master.HMaster
而启动regionSeriver主要是启动org.apache.hadoop.hbase.regionserver.HRegionServer里的main函数
zk 对应org.apache.hadoop.hbase.zookeeper.HQuorumPeer

主要结论:
HLog文件查看可以通过WALPrettyPrinter
HFIle可以通过类HFilePrettyPrinter查看
下面开始分析,第一个脚本start-hbase.sh

  1. usage="Usage: start-hbase.sh [--autostart-window-size <window size in hours>]\
  2. [--autostart-window-retry-limit <retry count limit for autostart>]\
  3. [autostart|start]"
  4. #获取当前目录名
  5. bin=`dirname "${BASH_SOURCE-$0}"`
  6. bin=`cd "$bin">/dev/null; pwd`
  7. # default autostart args value indicating infinite window size and no retry limit
  8. AUTOSTART_WINDOW_SIZE=0
  9. AUTOSTART_WINDOW_RETRY_LIMIT=0
  10. #加载配置
  11. . "$bin"/hbase-config.sh
  12. # start hbase daemons
  13. errCode=$
  14. if [ $errCode -ne 0 ]
  15. then
  16. exit $errCode
  17. fi
  18. if [ "$1" = "autostart" ]
  19. then
  20. commandToRun="--autostart-window-size ${AUTOSTART_WINDOW_SIZE} --autostart-window-retry-limit ${AUTOSTART_WINDOW_RETRY_LIMIT} autostart"
  21. else
  22. commandToRun="start"
  23. fi
  24. #判断模式
  25. # HBASE-6504 - only take the first line of the output in case verbose gc is on
  26. distMode=`$bin/hbase --config "$HBASE_CONF_DIR" org.apache.hadoop.hbase.util.HBaseConfTool hbase.cluster.distributed | head -n 1`
  27. //启动
  28. if [ "$distMode" == 'false' ]
  29. then
  30. "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}" $commandToRun master
  31. else
  32. "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" $commandToRun zookeeper
  33. "$bin"/hbase-daemon.sh --config "${HBASE_CONF_DIR}" $commandToRun master
  34. "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \
  35. --hosts "${HBASE_REGIONSERVERS}" $commandToRun regionserver
  36. "$bin"/hbase-daemons.sh --config "${HBASE_CONF_DIR}" \
  37. --hosts "${HBASE_BACKUP_MASTERS}" $commandToRun master-backup
  38. fi

配置类hbase-config.sh ,这个就是指明bin,conf,home,javahome,master等文件。
  1. ....
  2. # Allow alternate hbase conf dir location.
  3. HBASE_CONF_DIR="${HBASE_CONF_DIR:-$HBASE_HOME/conf}"
  4. # List of hbase regions servers.
  5. HBASE_REGIONSERVERS="${HBASE_REGIONSERVERS:-$HBASE_CONF_DIR/regionservers}"
  6. # List of hbase secondary masters.
  7. HBASE_BACKUP_MASTERS="${HBASE_BACKUP_MASTERS:-$HBASE_CONF_DIR/backup-masters}"
  8. if [ -n "$HBASE_JMX_BASE" ] && [ -z "$HBASE_JMX_OPTS" ]; then
  9. HBASE_JMX_OPTS="$HBASE_JMX_BASE"
  10. fi
  11. # Thrift JMX opts
  12. if [ -n "$HBASE_JMX_OPTS" ] && [ -z "$HBASE_THRIFT_JMX_OPTS" ]; then
  13. HBASE_THRIFT_JMX_OPTS="$HBASE_JMX_OPTS -Dcom.sun.management.jmxremote.port=10103"
  14. fi
  15. # Thrift opts
  16. if [ -z "$HBASE_THRIFT_OPTS" ]; then
  17. export HBASE_THRIFT_OPTS="$HBASE_THRIFT_JMX_OPTS"
  18. fi
  19. # REST JMX opts
  20. if [ -n "$HBASE_JMX_OPTS" ] && [ -z "$HBASE_REST_JMX_OPTS" ]; then
  21. HBASE_REST_JMX_OPTS="$HBASE_JMX_OPTS -Dcom.sun.management.jmxremote.port=10105"
  22. fi
  23. # REST opts
  24. if [ -z "$HBASE_REST_OPTS" ]; then
  25. export HBASE_REST_OPTS="$HBASE_REST_JMX_OPTS"
  26. fi
  27. # Source the hbase-env.sh. Will have JAVA_HOME defined.
  28. # HBASE-7817 - Source the hbase-env.sh only if it has not already been done. HBASE_ENV_INIT keeps track of it.
  29. if [ -z "$HBASE_ENV_INIT" ] && [ -f "${HBASE_CONF_DIR}/hbase-env.sh" ]; then
  30. . "${HBASE_CONF_DIR}/hbase-env.sh"
  31. export HBASE_ENV_INIT="true"
  32. fi
  33. # Verify if hbase has the mlock agent
  34. if [ "$HBASE_REGIONSERVER_MLOCK" = "true" ]; then
  35. MLOCK_AGENT="$HBASE_HOME/lib/native/libmlockall_agent.so"
  36. if [ ! -f "$MLOCK_AGENT" ]; then
  37. cat 1>&2 <<EOF
  38. Unable to find mlockall_agent, hbase must be compiled with -Pnative
  39. EOF
  40. exit 1
  41. fi
  42. if [ -z "$HBASE_REGIONSERVER_UID" ] || [ "$HBASE_REGIONSERVER_UID" == "$USER" ]; then
  43. HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -agentpath:$MLOCK_AGENT"
  44. else
  45. HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -agentpath:$MLOCK_AGENT=user=$HBASE_REGIONSERVER_UID"
  46. fi
  47. fi
  48. export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-4}
  49. # Now having JAVA_HOME defined is required
  50. if [ -z "$JAVA_HOME" ]; then
  51. cat 1>&2 <<EOF
  52. EOF
  53. exit 1
  54. fi
最后一个。hbase
主要根据参数启动不同的java对象。
  1. bin=`dirname "$0"`
  2. bin=`cd "$bin">/dev/null; pwd`
  3. # This will set HBASE_HOME, etc.
  4. . "$bin"/hbase-config.sh
  5. #判断cygwin
  6. cygwin=false
  7. case "`uname`" in
  8. CYGWIN*) cygwin=true;;
  9. esac
  10. #是否是开发环境
  11. # Detect if we are in hbase sources dir
  12. in_dev_env=false
  13. if [ -d "${HBASE_HOME}/target" ]; then
  14. in_dev_env=true
  15. fi
  16. #省去参数判断
  17. #获取命令
  18. # get arguments
  19. COMMAND=$1
  20. shift
  21. JAVA=$JAVA_HOME/bin/java
  22. #去除加载文件
  23. #加载jruby 文件
  24. # check if the commmand needs jruby
  25. declare -a jruby_cmds=("shell" "org.jruby.Main")
  26. for cmd in "${jruby_cmds[@]}"; do
  27. if [[ $cmd == "$COMMAND" ]]; then
  28. jruby_needed=true
  29. break
  30. fi
  31. done
  32. # the command needs jrub
  33. 计算命令执行那些东西
  34. # figure out which class to run
  35. if [ "$COMMAND" = "shell" ] ; then
  36. #find the hbase ruby sources
  37. if [ -d "$HBASE_HOME/lib/ruby" ]; then
  38. HBASE_OPTS="$HBASE_OPTS -Dhbase.ruby.sources=$HBASE_HOME/lib/ruby"
  39. else
  40. HBASE_OPTS="$HBASE_OPTS -Dhbase.ruby.sources=$HBASE_HOME/hbase-shell/src/main/ruby"
  41. fi
  42. HBASE_OPTS="$HBASE_OPTS $HBASE_SHELL_OPTS"
  43. CLASS="org.jruby.Main -X+O ${JRUBY_OPTS} ${HBASE_HOME}/bin/hirb.rb"
  44. elif [ "$COMMAND" = "hbck" ] ; then
  45. CLASS='org.apache.hadoop.hbase.util.HBaseFsck'
  46. # TODO remove old 'hlog' version
  47. elif [ "$COMMAND" = "hlog" -o "$COMMAND" = "wal" ] ; then
  48. CLASS='org.apache.hadoop.hbase.wal.WALPrettyPrinter'
  49. elif [ "$COMMAND" = "hfile" ] ; then
  50. CLASS='org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter'
  51. elif [ "$COMMAND" = "zkcli" ] ; then
  52. CLASS="org.apache.hadoop.hbase.zookeeper.ZooKeeperMainServer"
  53. elif [ "$COMMAND" = "backup" ] ; then
  54. CLASS='org.apache.hadoop.hbase.backup.BackupDriver'
  55. elif [ "$COMMAND" = "restore" ] ; then
  56. CLASS='org.apache.hadoop.hbase.backup.RestoreDriver'
  57. elif [ "$COMMAND" = "upgrade" ] ; then
  58. echo "This command was used to upgrade to HBase 0.96, it was removed in HBase 2.0.0."
  59. echo "Please follow the documentation at http://hbase.apache.org/book.html#upgrading."
  60. exit 1
  61. elif [ "$COMMAND" = "snapshot" ] ; then
  62. SUBCOMMAND=$1
  63. shift
  64. if [ "$SUBCOMMAND" = "create" ] ; then
  65. CLASS="org.apache.hadoop.hbase.snapshot.CreateSnapshot"
  66. elif [ "$SUBCOMMAND" = "info" ] ; then
  67. CLASS="org.apache.hadoop.hbase.snapshot.SnapshotInfo"
  68. elif [ "$SUBCOMMAND" = "export" ] ; then
  69. CLASS="org.apache.hadoop.hbase.snapshot.ExportSnapshot"
  70. else
  71. echo "Usage: hbase [<options>] snapshot <subcommand> [<args>]"
  72. echo "$options_string"
  73. echo ""
  74. echo "Subcommands:"
  75. echo " create Create a new snapshot of a table"
  76. echo " info Tool for dumping snapshot information"
  77. echo " export Export an existing snapshot"
  78. exit 1
  79. fi
  80. elif [ "$COMMAND" = "master" ] ; then
  81. CLASS='org.apache.hadoop.hbase.master.HMaster'
  82. if [ "$1" != "stop" ] && [ "$1" != "clear" ] ; then
  83. HBASE_OPTS="$HBASE_OPTS $HBASE_MASTER_OPTS"
  84. fi
  85. elif [ "$COMMAND" = "regionserver" ] ; then
  86. CLASS='org.apache.hadoop.hbase.regionserver.HRegionServer'
  87. if [ "$1" != "stop" ] ; then
  88. HBASE_OPTS="$HBASE_OPTS $HBASE_REGIONSERVER_OPTS"
  89. fi
  90. elif [ "$COMMAND" = "thrift" ] ; then
  91. CLASS='org.apache.hadoop.hbase.thrift.ThriftServer'
  92. if [ "$1" != "stop" ] ; then
  93. HBASE_OPTS="$HBASE_OPTS $HBASE_THRIFT_OPTS"
  94. fi
  95. elif [ "$COMMAND" = "thrift2" ] ; then
  96. CLASS='org.apache.hadoop.hbase.thrift2.ThriftServer'
  97. if [ "$1" != "stop" ] ; then
  98. HBASE_OPTS="$HBASE_OPTS $HBASE_THRIFT_OPTS"
  99. fi
  100. elif [ "$COMMAND" = "rest" ] ; then
  101. CLASS='org.apache.hadoop.hbase.rest.RESTServer'
  102. if [ "$1" != "stop" ] ; then
  103. HBASE_OPTS="$HBASE_OPTS $HBASE_REST_OPTS"
  104. fi
  105. elif [ "$COMMAND" = "zookeeper" ] ; then
  106. CLASS='org.apache.hadoop.hbase.zookeeper.HQuorumPeer'
  107. if [ "$1" != "stop" ] ; then
  108. HBASE_OPTS="$HBASE_OPTS $HBASE_ZOOKEEPER_OPTS"
  109. fi
  110. elif [ "$COMMAND" = "clean" ] ; then
  111. case $1 in
  112. --cleanZk|--cleanHdfs|--cleanAll)
  113. matches="yes" ;;
  114. *) ;;
  115. esac
  116. if [ $# -ne 1 -o "$matches" = "" ]; then
  117. echo "Usage: hbase clean (--cleanZk|--cleanHdfs|--cleanAll)"
  118. echo "Options: "
  119. echo " --cleanZk cleans hbase related data from zookeeper."
  120. echo " --cleanHdfs cleans hbase related data from hdfs."
  121. echo " --cleanAll cleans hbase related data from both zookeeper and hdfs."
  122. exit 1;
  123. fi
  124. "$bin"/hbase-cleanup.sh --config ${HBASE_CONF_DIR} $@
  125. exit $
  126. elif [ "$COMMAND" = "mapredcp" ] ; then
  127. CLASS='org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool'
  128. elif [ "$COMMAND" = "classpath" ] ; then
  129. echo $CLASSPATH
  130. exit 0
  131. elif [ "$COMMAND" = "pe" ] ; then
  132. CLASS='org.apache.hadoop.hbase.Performanceeva luation'
  133. HBASE_OPTS="$HBASE_OPTS $HBASE_PE_OPTS"
  134. elif [ "$COMMAND" = "ltt" ] ; then
  135. CLASS='org.apache.hadoop.hbase.util.LoadTestTool'
  136. HBASE_OPTS="$HBASE_OPTS $HBASE_LTT_OPTS"
  137. elif [ "$COMMAND" = "canary" ] ; then
  138. CLASS='org.apache.hadoop.hbase.tool.Canary'
  139. HBASE_OPTS="$HBASE_OPTS $HBASE_CANARY_OPTS"
  140. elif [ "$COMMAND" = "version" ] ; then
  141. CLASS='org.apache.hadoop.hbase.util.VersionInfo'
  142. else
  143. CLASS=$COMMAND
  144. fi
  145. # Have JVM dump heap if we run out of memory. Files will be 'launch directory'
  146. # and are named like the following: java_pid21612.hprof. Apparently it doesn't
  147. # 'cost' to have this flag enabled. Its a 1.6 flag only. See:
  148. # http://blogs.sun.com/alanb/entry/outofmemoryerror_looks_a_bit_better
  149. HBASE_OPTS="$HBASE_OPTS -Dhbase.log.dir=$HBASE_LOG_DIR"
  150. HBASE_OPTS="$HBASE_OPTS -Dhbase.log.file=$HBASE_LOGFILE"
  151. HBASE_OPTS="$HBASE_OPTS -Dhbase.home.dir=$HBASE_HOME"
  152. HBASE_OPTS="$HBASE_OPTS -Dhbase.id.str=$HBASE_IDENT_STRING"
  153. HBASE_OPTS="$HBASE_OPTS -Dhbase.root.logger=${HBASE_ROOT_LOGGER:-INFO,console}"
  154. if [ "x$JAVA_LIBRARY_PATH" != "x" ]; then
  155. HBASE_OPTS="$HBASE_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH"
  156. export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$JAVA_LIBRARY_PATH"
  157. fi
  158. # Enable security logging on the master and regionserver only
  159. if [ "$COMMAND" = "master" ] || [ "$COMMAND" = "regionserver" ]; then
  160. HBASE_OPTS="$HBASE_OPTS -Dhbase.security.logger=${HBASE_SECURITY_LOGGER:-INFO,RFAS}"
  161. else
  162. HBASE_OPTS="$HBASE_OPTS -Dhbase.security.logger=${HBASE_SECURITY_LOGGER:-INFO,NullAppender}"
  163. fi
  164. HEAP_SETTINGS="$JAVA_HEAP_MAX $JAVA_OFFHEAP_MAX"
  165. # Exec unless HBASE_NOEXEC is set.
  166. export CLASSPATH
  167. if [ "${HBASE_NOEXEC}" != "" ]; then
  168. "$JAVA" -Dproc_$COMMAND -XX:OnOutOfMemoryError="kill -9 %p" $HEAP_SETTINGS $HBASE_OPTS $CLASS "$@"
  169. else
  170. exec "$JAVA" -Dproc_$COMMAND -XX:OnOutOfMemoryError="kill -9 %p" $HEAP_SETTINGS $HBASE_OPTS $CLASS "$@"
  171. fi
最后执行Java类
COMMAND : start/stop/restart
OnOutOfMemoryError : 当内存溢出时直接杀死进程
HEAP_SETTINGS :设置最大堆,
CLASS :这个就是org.apache.hadoop.hbase.regionserver.HRegionServer 等。

最后执行Java类
COMMAND : start/stop/restart
OnOutOfMemoryError : 当内存溢出时直接杀死进程
HEAP_SETTINGS :设置最大堆,
CLASS :这个就是org.apache.hadoop.hbase.regionserver.HRegionServer 等。

从上面可以看出,通过
HFilePrettyPrinter 可以参考hfile内容是什么。
WALPrettyPrinter 可以打开HLOG文件,可以

而且hbase shell 是启动的ruby 文件。

HMaster的启动入口
public static void main(String [] args) {
  VersionInfo.logVersion();
  new HMasterCommandLine(HMaster.class).doMain(args);
}
HRegionSerive的启动入口

/**
 * @see org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine
 */
public static void main(String[] args) throws Exception {
  VersionInfo.logVersion();
  Configuration conf = HBaseConfiguration.create();
  @SuppressWarnings("unchecked")
  Class< extends HRegionServer> regionServerClass = (Class< extends HRegionServer>) conf
      .getClass(HConstants.REGION_SERVER_IMPL, HRegionServer.class);

  new HRegionServerCommandLine(regionServerClass).doMain(args);
}

后面将继续将HMaster和HRegionService启动过程

未完待续。。。


】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇HBase---使用hbase shell和Get对.. 下一篇用Hbase实现二级索引

最新文章

热门文章

Hot 文章

Python

C 语言

C++基础

大数据基础

linux编程基础

C/C++面试题目