设为首页 加入收藏

TOP

Spark伪分布式安装(一)
2019-04-15 01:05:51 】 浏览:35
Tags:Spark 分布式 安装
版权声明:小兄弟,慢慢熬吧...... https://blog.csdn.net/u013850277/article/details/73648108

笔者是在已安装好的伪分布式Hadoop环境下安装的Spark。虚拟机环境为:centos7。

Hadoop版本号为:

[centosm@centosm spark]$ hadoop version
Hadoop 2.7.2

一、Spark伪分布式安装

1、确定hadoop环境已安装好

2、下载spark
http://spark.apache.org/downloads.html
这里写图片描述

3、解压spark压缩包并重命名解压后的文件名为spark

     tar -zvxf  spark
 
			
subst">-
2.1.1-bin-hadoop2.7.tgz

4、修改配置文件

     cd /usr/local/spark
     cp ./conf/spark-env.sh.template ./conf/spark-env.sh

     spark-env.sh 文件后追加如下语句

vi spark-env.sh
export SPARK_DIST_CLASSPATH=$(/home/centosm/hadoopM/bin/hadoop classpath)

5、启动spark,运行时出现下列问题

这里写图片描述

解决:

vi  spark-config.sh
export JAVA_HOME=/usr/java/jdk1.8.0_91

6、运行spark自带的一些小例子,如下计算 π 的近似值

[centosm@centosm spark]$ ./bin/run-example SparkPi 2>&1 | grep "Pi is roughly"
Pi is roughly 3.1428357141785708

7、进入spark交互模式

[centosm@centosm spark]$ ./bin/spark-shell            
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/centosm/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/centosm/hadoopM/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/06/22 14:51:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/06/22 14:52:15 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.22.125:4040
Spark context available as 'sc' (master = local[*], app id = local-1498114310652).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_91)
Type in expressions to have them eva luated.
Type :help for more information.

scala> 

在交互模式的情启动的情况可以进入web页面查看相关信息,如下:

这里写图片描述

8、yum 安装sbt

[centosm@centosm test]$ curl https://bintray.com/sbt/rpm/rpm > bintray-sbt-rpm.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   159    0   159    0     0     23      0 --:--:--  0:00:06 --:--:--    37
[centosm@centosm test]$
[centosm@centosm test]$ sudo mv bintray-sbt-rpm.repo /etc/yum.repos.d/
[root@centosm test]# yum install sbt
[centosm@centosm bin]$ sbt --version
Getting org.scala-sbt sbt 0.13.15  (this may take some time)...

编程开发网
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇spark常见问题 下一篇Spark运行原理【史上最详细】

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容:

array(4) { ["type"]=> int(8) ["message"]=> string(24) "Undefined variable: jobs" ["file"]=> string(32) "/mnt/wp/cppentry/do/bencandy.php" ["line"]=> int(217) }