配置&使用Spark History Server
以默认配置的方式启动spark history server:
cd $SPARK_HOME/sbin
start-history-server.sh
报错
starting org.apache.spark.deploy.history.HistoryServer, logging to /home/spark/software/source/compile/deploy_spark/sbin/../logs/spark-spark-org.apache.spark.deploy.history.HistoryServer-1-hadoop000.out
failed to launch org.apache.spark.deploy.history.HistoryServer:
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:44)
... 6 more
[root@biluos logs]# /opt/moudles/spark-2.2.0-bin-hadoop2.7/sbin/start-history-server.sh hdfs://mycluster:8020/spark_job_history
starting org.apache.spark.deploy.history.HistoryServer, logging to /opt/moudles/spark-2.2.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-biluos.com.out
[root@biluos logs]# cat spark-root-org.apache.spark.deploy.history.HistoryServer-1-biluos.com.out SparkCommand: /opt/moudles/jdk1.8.0_121/bin/java -cp /opt/moudles/spark-2.2.0-bin-hadoop2.7/conf/:/opt/moudles/spark-2.2.0-bin-hadoop2.7/jars/*:/opt/moudles/hadoop-2.7.3/etc/hadoop/ -Xmx1g org.apache.spark.deploy.history.HistoryServerhdfs://mycluster:8020/spark_job_history
========================================
17/08/0303:22:18INFOHistoryServer:Started daemon with process name:2666@biluos.com
17/08/0303:22:18INFOSignalUtils:Registered signal handler forTERM17/08/0303:22:18INFOSignalUtils:Registered signal handler forHUP17/08/0303:22:18INFOSignalUtils:Registered signal handler forINT17/08/0303:22:18WARNHistoryServerArguments:Setting log directory through the command line is deprecated as of Spark1.1.0. Please set this through spark.history.fs.logDirectory instead.
17/08/0303:22:19WARNNativeCodeLoader:Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/0303:22:19INFOSecurityManager:Changing view acls to: root
17/08/0303:22:19INFOSecurityManager:Changing modify acls to: root
17/08/0303:22:19INFOSecurityManager:Changing view acls groups to:17/08/0303:22:19INFOSecurityManager:Changing modify acls groups to:17/08/0303:22:19INFOSecurityManager:SecurityManager: authentication disabled; ui acls disabled; users with view permissions:Set(root); groups with view permissions:Set(); users with modify permissions:Set(root); groups with modify permissions:Set()
17/08/0303:22:19INFOFsHistoryProvider:History server ui acls disabled; users with admin permissions: ; groups with admin permissions
Exceptionin thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:278)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Causedby: java.io.FileNotFoundException:Log directory specified does notexist:hdfs://mycluster:8020/spark_job_history
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:214)
at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:160)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:156)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:78)
... 6 more
Causedby: java.io.FileNotFoundException:File does notexist:hdfs://mycluster:8020/spark_job_history
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:204)
... 9 more
解决方法
[root@biluos logs]# hdfs dfs -mkdir /spark_job_history
重新启动不报错了