版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/u012848709/article/details/85462762
楔子
Spark 提交作业找不到jar
ERROR ClientEndpoint:70 - Exception from cluster was: java.nio.file.NoSuchFileException
出错提示
[root@had4 spark240]# ./bin/spark-submit \
> --master spark://had1:7077 \
> --deploy-mode cluster \
> --supervise \
> --class cn.sxt.spark.core.WC \
> wc.jar
2018-12-26 17:01:06 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-26 17:01:13 ERROR ClientEndpoint:70 - Exception from cluster was: java.nio.file.NoSuchFileException: /opt/spark240/wc.jar
java.nio.file.NoSuchFileException: /opt/spark240/wc.jar
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
at java.nio.file.Files.copy(Files.java:1274)
环境
hadoop是基于3.1.1
spark是基于 2.4.0版本
提交
采用hdfs存放文件 不知道为何存放在本地有问题
./bin/spark-submit \
--master spark://had1:7077 \
--deploy-mode cluster \
--class cn.sxt.spark.core.WC hdfs://had2/file/wc.jar
找不到jar
测试了2次 分别如下
-----成功的提交
./bin/spark-submit --class cn.sxt.spark.core.WC --master spark://had1:7077 wc.jar
-- 失败的提交
./bin/spark-submit --class cn.sxt.spark.core.WC --master spark://had1:7077 --deploy-mode cluster wc.jar
第一次是standalone -client 提交 ,结果提交成功了
第二次是 standalone-cluster提交,结果提示找不到jar了。spark master 节点和worker 节点 都放置了jar文件,结果成功了
。或许是因为集群运行,需要在各个节点上都放置jar文件
其他提交实例
官网提交实例参考
hadoop一个旧版本中文文档
# Run application locally on 8 cores
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master local[8] \
/path/to/examples.jar \
100
# Run on a Spark standalone cluster in client deploy mode
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://207.184.161.138:7077 \
--executor-memory 20G \
--total-executor-cores 100 \
/path/to/examples.jar \
1000
# Run on a Spark standalone cluster in cluster deploy mode with supervise
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://207.184.161.138:7077 \
--deploy-mode cluster \
--supervise \
--executor-memory 20G \
--total-executor-cores 100 \
/path/to/examples.jar \
1000
# Run on a YARN cluster
export HADOOP_CONF_DIR=XXX
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master yarn \
--deploy-mode cluster \ # can be client for client mode
--executor-memory 20G \
--num-executors 50 \
/path/to/examples.jar \
1000
# Run a Python application on a Spark standalone cluster
./bin/spark-submit \
--master spark://207.184.161.138:7077 \
examples/src/main/python/pi.py \
1000
# Run on a Mesos cluster in cluster deploy mode with supervise
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master mesos://207.184.161.138:7077 \
--deploy-mode cluster \
--supervise \
--executor-memory 20G \
--total-executor-cores 100 \
http://path/to/examples.jar \
1000
# Run on a Kubernetes cluster in cluster deploy mode
./bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master k8s://xx.yy.zz.ww:443 \
--deploy-mode cluster \
--executor-memory 20G \
--num-executors 50 \
http://path/to/examples.jar \
1000