设为首页 加入收藏

TOP

Spark安装与配置
2019-05-15 01:05:32 】 浏览:7
Tags:Spark 安装 配置
版权声明:本文为博主原创文章,转载请声明原创网址。 https://blog.csdn.net/lagoon_lala/article/details/89713073

在主节点jps查看进程

start-all.sh启动能够开启ResourceManager

Jps

97505 Jps

96960 NameNode

30195 Application

97246 ResourceManager

Windows-java使用hadoop配置

下载Hadoopwindows,设置环境变量为hadoop目录-bin-sbin目录

下载hadoop-common-bin替换bin

winutils.exehadoop.dll

修改tpotspark工程中resourcemongoConfig

Tpotspark 工程install后打jar包上传到hadoop线上目录

打包遇到jre不匹配问题:Eclipcejre改成jdk下的jre

打包命令mvn -Dmaven.test.skip=true assembly:assembly

上传jar包:

spark-submit --master yarn --class com.tlbcc.spark.App hdfs://master:9000/spark/sparktrain-1.0.jar hdfs://master:9000/flume/suricata/19-02-11/2019-02-11.1549877608019 suricata

spark-submit 提交任务及参数说明:

https://www.cnblogs.com/weiweifeng/p/8073553.html

spark-submit 可以提交任务到 spark 集群执行,也可以提交到 hadoop yarn 集群执行

搭建spark

Hadoop()spark环境搭建:https://blog.csdn.net/mingyunxiaohai/article/details/80642907

Hadoop2.7.3+Spark2.1.0 完全分布式环境 搭建全过程 :

https://www.cnblogs.com/purstar/p/6293605.html

https://www.cnblogs.com/tovin/p/3820979.html

不知道hadoop版本会不会对spark版本有影响,试spark2.1.0

2.Scala2.12.2环境搭建:

在集群所有节点下载并解压spark的安装包:

su hdp

cd /home/hdp

wget http://d3kbcqa49mib13.cloudfront.net/spark-1.0.0-bin-hadoop2.tgz

sudo mv /home/hdp/spark-1.0.0-bin-hadoop2.tgz /usr/local/

cd /usr/local/

sudo tar zxvf spark-1.0.0-bin-hadoop2.tgz

sudo ln -s spark-1.0.0-bin-hadoop2 spark

sudo chown -R hdp:hdp spark-1.0.0-bin-hadoop2

sudo rm -rf spark-1.0.0-bin-hadoop2.tgz #执行后spark: broken symbolic link to spark-2.1.0-bin-hadoop2.7

执行

wget http://d3kbcqa49mib13.cloudfront.net/spark-2.1.0-bin-hadoop2.7.tgz

sudo tar zxvf spark-2.1.0-bin-hadoop2.7.tgz

sudo ln -s spark-2.1.0-bin-hadoop2.7 spark

user@Hadoop1Server:~/spark$ sudo chown -R user:user spark-2.1.0-bin-hadoop2.7

Spark部署

spark standalone

node01master节点,node02node03slave节点安装为例说明:

1、修改集群所有节点spark环境配置文件
cd /usr/local/spark/conf/
mv spark-env.sh.template spark-env.sh
vim spark-env.sh 添加如下内容:      

    上面参数可以根据机器实际资源情况进行设置其中:
SPARK_WORKER_CORES表示每个Worker进程使用core数目
SPARK_WORKER_MEMORY表示每个Worker进程使用内存
SPARK_WORKER_INSTANCES表示每台机器Worker数目

执行:

cd spark/conf/
sudo cp spark-env.sh.template spark-env.sh

vim spark-env.sh

export SPARK_MASTER_IP=10.2.68.104

export SPARK_WORKER_CORES=1

export SPARK_WORKER_MEMORY=500m

export SPARK_WORKER_INSTANCES=1

2、启动集群

/usr/local/spark/sbin/start-all.sh

执行:user@Hadoop1Server:~/spark/spark$ sbin/start-all.sh

报错:localhost: JAVA_HOME is not set

解决:在spark-config.sh里加上

export JAVA_HOME=/path/to/java

执行:

user@Hadoop1Server:~/spark/spark/sbin$ vim spark-config.sh

export JAVA_HOME=/home/user/jdk1.8.0_171

报错:

user@Hadoop1Server:~/spark/spark$ sbin/start-all.sh

org.apache.spark.deploy.master.Master running as process 121242. Stop it first.

localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/user/spark/spark/logs/spark-user-org.apache.spark.deploy.worker.Worker-1-Hadoop1Server.out

解决:

sbin/stop-all.sh

集群web ui

http://10.2.68.104:8080/查看集群管理页面

spark on yarn模式

SparkOnYarn专题:https://blog.csdn.net/qq_21439395/article/details/80678372

Spark On Yarn安装使用:https://blog.csdn.net/lbship/article/details/82854524

搭建yarn

Web查看yarn:8088端口

修改配置文件:yarn-site.xml

位置:user@Hadoop1Server:~/hadoop/etc/hadoop

启动yarn集群:(注:该命令应该在resourcemanager所在的机器上执行)

# start-yarn.sh

停止:# stop-yarn.sh

验证:用jps检查yarn的进程,用web浏览器查看yarnweb控制台

启动完成后,可以在windows上用浏览器访问resourcemanagerweb端口:

http://hdp-01:8088

yarn集群的补充配置:

补充配置一:

yarn默认情况下,只根据内存调度资源,所以sparkon yarn运行的时候,即使通过--executor-cores指定vcore个数为N,但是在yarn的资源管理页面上看到使用的vcore个数还是1. 相关配置在capacity-scheduler.xml 文件:

user@Hadoop1Server:~/hadoop/etc/hadoop$ vim capacity-scheduler.xml

<property>

<name>yarn.scheduler.capacity.resource-calculator</name>

<!--<value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value> -->

<value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>

</property>

补充配置二:

修改所有yarn节点的yarn-site.xml,在该文件中添加如下配置

如果不配置这两个选项,在spark-on-yarnclient模式下,可能会报错,错误如下:

user@Hadoop1Server:~/hadoop/etc/hadoop$ vim yarn-site.xml

<property>

<name>yarn.nodemanager.pmem-check-enabled</name>

<value>false</value>

</property>

<property>

<name>yarn.nodemanager.vmem-check-enabled</name>

<value>false</value>

</property>

参数说明:

yarn.nodemanager.pmem-check-enabled

是否启动一个线程检查每个任务正使用的物理内存量,如果任务超出分配值,则直接将其杀掉,默认是true

yarn.nodemanager.vmem-check-enabled

是否启动一个线程检查每个任务正使用的虚拟内存量,如果任务超出分配值,则直接将其杀掉,默认是true

配置修改完成之后,把配置文件分发到各台节点上:

# cd /root/apps/hadoop/etc/hadoop

[root@hdp-01 hadoop]# for i in 2 3 ;do scp capacity-scheduler.xml yarn-site.xml hdp-0$i:`pwd`;done

执行:

user@Hadoop1Server:~$ scp ~/hadoop/etc/hadoop/yarn-site.xml user@Hadoop3Server:~/hadoop/etc/hadoop/yarn-site.xml

user@Hadoop1Server:~$ scp ~/hadoop/etc/hadoop/yarn-site.xml user@Hadoop2Server:~/hadoop/etc/hadoop/yarn-site.xml

scp ~/hadoop/etc/hadoop/capacity-scheduler.xml user@Hadoop2Server:~/hadoop/etc/hadoop/capacity-scheduler.xml

scp ~/hadoop/etc/hadoop/capacity-scheduler.xml user@Hadoop3Server:~/hadoop/etc/hadoop/capacity-scheduler.xml

Spark

spark-env.sh配置:https://blog.csdn.net/weixin_42186022/article/details/84942998

yarnhadoop中的一个组件,是一个统一的资源调度平台。

spark onyarn,就是把spark任务提交到yarn 集群上运行。

那么提交spark任务的地方,就是客户端。所以客户端一台即可。但需要保证客户端可以正常连接到hdfs集群和yarn集群。

spark配置:官方文档http://spark.apache.org/docs/latest/running-on-yarn.html

下载spark安装包,上传并解压到/root/apps目录下。

修改spark的配置文件,只需要修改conf目录下的spark-env.sh 配置文件即可。

spark-env.sh中配置

export JAVA_HOME=/usr/local/jdk1.8.0_131

export HADOOP_CONF_DIR=/root/apps/hadoop/etc/hadoop

执行:

export JAVA_HOME=/home/user/jdk1.8.0_171

export HADOOP_HOME=/home/user/hadoop/etc/hadoop

可以使用HADOOP_CONF_DIR或者YARN_CONF_DIR

提交spark任务的地方,就是客户端,所以配置一台机器即可

启动hdfsstart-dfs.sh

启动yarn: start-yarn.sh

如果用stop-all.sh会显示:

This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh

Client Driver部署:

1、下载sparkhadoop安装包
    参照系统环境配置部分进行设置
2、修改配置文件
hadoop配置文件使用与集群一致的文件
cd /usr/local/spark
vim conf/spark-env.sh添加内容

3spark测试程序
/usr/local/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster /usr/local/spark/lib/spark-examples-1.0.0-hadoop2.2.0.jar

测试spark-shell

spark-shell 命令使用:https://blog.csdn.net/wawa8899/article/details/81016029

1. spark-shell 使用帮助[hadoop@hadoop01 ~]$ cd app/spark-2.2.0-bin-2.6.0-cdh5.7.0/bin

[hadoop@hadoop01 bin]$ ./spark-shell --help

Usage: ./bin/spark-shell [options]

spark-shell 底层也是调用spark-submit进行作业的提交的(源码查看: $SPARK_HOME/bin/spark-shell

其他的诸如spark-sqlsparkR等底层也都是由spark-submit来进行作业提交的。

执行:

user@Hadoop1Server:~/spark/spark/bin$ ./spark-shell

spark-submit

spark-submit 提交任务及参数说明https://www.cnblogs.com/weiweifeng/p/8073553.html

Spark-submit提交任务到集群:https://blog.csdn.net/hellozhxy/article/details/80483376

spark-submit --master yarn --class com.tlbcc.spark.App hdfs://master:9000/spark/sparktrain-1.0.jar hdfs://master:9000/flume/suricata/19-02-11/2019-02-11.1549877608019 suricata

启动hdfs,创建spark目录,上传jar

hdfs dfs -put /home/user/spark/sparktrain/sparktrain-1.0.jar /spark

修改命令中的suricata目录为已存在的

执行:

hdfs dfs -mkdir /spark

首先启动集群,然后客户端来到spark-submit目录:/app/hadoop/spark131/bin

执行:

user@Hadoop1Server:~$ cd spark/spark/bin/

user@Hadoop1Server:~/spark/spark/bin$./spark-submit --master yarn --class com.tlbcc.spark.App hdfs://master:9000/spark/sparktrain-1.0.jar hdfs://master:9000/flume/suricata/19-03-08/2019-03-08.1552051184606 suricata

报错:

Exception in thread "main" java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.

at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:256)

at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:233)

at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

解决:https://blog.csdn.net/strongyoung88/article/details/52201622

解决方法:
编辑$SPARK_HOME/conf/spark-env.sh文件

加入以下行:

HADOOP_CONF_DIR=/home/hadoop/hadoop-2.4.0/etc/hadoop/

执行:

export HADOOP_CONF_DIR=/home/user/hadoop/etc/hadoop/

报错:

Warning: Skip remote jar hdfs://master:9000/spark/sparktrain-1.0.jar.

java.lang.ClassNotFoundException: com.tlbcc.spark.App

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.spark.util.Utils$.classForName(Utils.scala:229)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:695)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

解决:

hosts中添加masteripping

手动执行jar

user@Hadoop1Server:~/spark/sparktrain$ java -jar sparktrain-1.0.jar hdfs://master:9000/flume/suricata/19-03-08/2019-03-08.1552051184606 suricata

报错:

Exception in thread "main" java.lang.ExceptionInInitializerError

at com.tlbcc.spark.App.main(App.java:38)

Caused by: java.lang.NullPointerException

at com.tlbcc.spark.SuricataHandler.<clinit>(SuricataHandler.java:26)

... 1 more

尝试建立本地.json文件测试

cd ~/spark/sparktrain

user@Hadoop1Server:~/spark/sparktrain$ java -jar sparktrain-1.0.jar /home/user/spark/sparktrain/eve.json suricata

没有区别,尝试在eclipse运行:

报错:

Exception in monitor thread while connecting to server 10.2.192.238:27017

com.mongodb.MongoSocketOpenException: Exception opening socket

at com.mongodb.internal.connection.SocketChannelStream.open(SocketChannelStream.java:60)

at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:126)

at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.ConnectException: Connection refused: no further information

开启mongodbeclipse不报错,尝试在windows命令行运行

E:\studyMaterial\学习\其他\活动项目\大数据工作室\Xmanager\spark\本地>java -jar sparktrain-1.0.jar E:\studyMaterial\work\spark\tpotsprak\eve.json suricata

无法执行

可能是spark没有获取到hadoop上的jar包,尝试获取,能够获取

转换成root用户,相同报错

java.lang.ClassNotFoundException

spark的安装配置时,我们在SPARK_HOME/conf/spark-defaults.conf文件中配置了如下的参数,spark on yarn模式,默认情况下会读取spark本地的jar包(再jars目录下)分配到yarncontainers

spark.yarn.jars=hdfs://c1-psc1-VB:9000/user/bd/spark/sparkjar/*`

参考stackoverflow.https://stackoverflow.com/questions/24206536/spark-java-appilcation-java-lang-classnotfoundexception

You need to deliver the jar with your job to the workers. To do that, have maven build a jar and add that jar to the context:

conf.setJars(new String[]{"path/to/jar/Sample.jar"}); [*]

You must include your jar in the worker's classpath. You can do this in two ways:

Using the SparkContext method addJar (you can review the documentation in this page http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext)

Adding the jar in the lib directory in your spark distribution.

The first one is the recommended method.

This can also happen if you do not specify the full package name when using spark-submit command line. If your main method for the application is in test.spark.SimpleApp then the command line needs to look something like this: 提交命令行时未指定完整的包名称, 也会发生这种情况。如果应用程序的主要方法位于 test.spark.SimpleApp 则命令行需要如下所示:

./bin/spark-submit --class "test.spark.SimpleApp" --master local[2] /path_to_project/target/spark_testing-1.0-SNAPSHOT.jar

Adding just --class "SimpleApp" will fail with ClassNotFoundException.

参考:

https://stackoverflow.com/questions/23408118/apache-spark-java-lang-classnotfoundexception

https://stackoverflow.com/questions/29511594/java-lang-classnotfoundexception-when-i-use-spark-submit-with-a-new-class-name

命令应该没错,还有一种可能就是jar打包有问题

目前,建议先拉一下最新代码

修改ip重新打一下包

resource中

host=10.2.192.238

这次代码已经很全了,除了conpot暂时没法用,其他的没问题了

然后代码想本地执行的话:把handler类中的master("local[2]")取消注释一下,每个handler里面有两个

打包完放到hdfs后用这个执行测一下,内容改成自己的

spark-submit --master yarn --class com.tlbcc.spark.App hdfs://master:9000/spark/sparktrain-1.0.jar hdfs://master:9000/flume/suricata/19-02-11/2019-02-11.1549877608019 suricata

在线执行

上传

hdfs dfs -put /home/user/spark/sparktrain/sparktrain-1.0.jar /spark

删除

hdfs dfs -rm -r /spark/sparktrain-1.0.jar

提交

user@Hadoop1Server:~$ cd home/user/spark/spark/bin/

user@Hadoop1Server:~/spark/spark/bin$./spark-submit --master yarn --class com.tlbcc.spark.App hdfs://master:9000/spark/sparktrain-1.0.jar hdfs://master:9000/flume/suricata/19-03-08/2019-03-08.1552051184606 suricata

提交命令行时未指定完整的包名称, 也会发生这种情况。如果应用程序的主要方法位于 test.spark.SimpleApp 则命令行需要如下所示:

./bin/spark-submit --class "test.spark.SimpleApp" --master local[2] /path_to_project/target/spark_testing-1.0-SNAPSHOT.jar

本地执行:

cd ~/spark/sparktrain

user@Hadoop1Server:~/spark/sparktrain$ java -jar sparktrain-1.0.jar hdfs://master:9000/flume/suricata/19-03-08/2019-03-08.1552051184606 suricata

成功:

user@Hadoop1Server:~/spark/spark/bin$./spark-submit --master yarn --class com.tlbcc.spark.App ~/spark/sparktrain/sparktrain-1.0.jar hdfs://master:9000/flume/suricata/19-03-08/2019-03-08.1552051184606 suricata

显示:

19/04/29 14:58:37 INFO driver.connection: Opened connection [connectionId{localValue:3, serverValue:3}] to 10.2.192.238:27017

19/04/29 14:58:37 INFO driver.cluster: Monitor thread successfully connected to server with description ServerDescription{address=10.2.192.238:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 0]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=null, roundTripTimeNanos=1678182}

19/04/29 14:58:37 INFO driver.connection: Opened connection [connectionId{localValue:4, serverValue:4}] to 10.2.192.238:27017

19/04/29 14:58:37 INFO spark.App: 任务执行完毕.

19/04/29 14:58:37 INFO spark.SparkContext: Invoking stop() from shutdown hook

19/04/29 14:58:37 INFO server.ServerConnector: Stopped ServerConnector@9f46d94{HTTP/1.1}{0.0.0.0:4040}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@d78795{/stages/stage/kill,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@46074492{/jobs/job/kill,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@64da2a7{/api,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@c9413d8{/,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@32fe9d0a{/static,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@37eeec90{/executors/threadDump/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@724f138e{/executors/threadDump,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@cf65451{/executors/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@f415a95{/executors,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@114a85c2{/environment/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@12cd9150{/environment,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@47289387{/storage/rdd/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1950e8a6{/storage/rdd,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@35d6ca49{/storage/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@56e07a08{/storage,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5990e6c5{/stages/pool/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4a9cc6cb{/stages/pool,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@142eef62{/stages/stage/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@517bd097{/stages/stage,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@50687efb{/stages/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@66971f6b{/stages,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3bffddff{/jobs/job/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1601e47{/jobs/job,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@66ea1466{/jobs/json,null,UNAVAILABLE}

19/04/29 14:58:37 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5471388b{/jobs,null,UNAVAILABLE}

19/04/29 14:58:37 INFO ui.SparkUI: Stopped Spark web UI at http://10.2.68.104:4040

19/04/29 14:58:37 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

19/04/29 14:58:37 INFO memory.MemoryStore: MemoryStore cleared

19/04/29 14:58:37 INFO storage.BlockManager: BlockManager stopped

19/04/29 14:58:37 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

19/04/29 14:58:37 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

19/04/29 14:58:37 INFO spark.SparkContext: Successfully stopped SparkContext

19/04/29 14:58:37 INFO util.ShutdownHookManager: Shutdown hook called

19/04/29 14:58:37 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2280070d-9ffc-4526-a291-f8fc42c03d30

Jar包本地可以运行,hadoop上无法运行

启动spark

start-all.sh

user@Hadoop1Server:~$ ~/spark/spark/sbin/start-all.sh


编程开发网
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇Spark入门梳理1 下一篇Spark提交任务的方式

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容:

array(4) { ["type"]=> int(8) ["message"]=> string(24) "Undefined variable: jobs" ["file"]=> string(32) "/mnt/wp/cppentry/do/bencandy.php" ["line"]=> int(214) }