原文地址:http://blog.csdn.net/yangning5850/article/details/9143151
Spark——Lightning-Fast Cluster Computing,这是Spark 官方logo的内容,让人很期待它的计算速度是否真的如此之快。
Spark是由UC Berkeley AMPLab开发的,一种类似Hadoop MapReduce的系统,但是在读写速度上面都有很快的速度。大家都知道Hadoop框架I/O开销是很大的,基于Hadoop的MapReduce运算很多是基于磁盘的。Spark设计的初衷,就是为那些需多次内存迭代的运算设计的,在机器学习和数据挖掘中,有很多算法需要用到这样的特性。所以,Spark支持内存环境的集群运算(不清楚它如何存放大量数据,待进一步研究),并且提供了Shark——一种完全兼容Hive的数据仓库系统,但是要比Hive快100多倍。
UC Berkeley最伟大的地方在于有很多研究成果都贡献给了开源社区,Spark同样是一个开源项目,下面就让我们试着安装一下吧。
我的测试环境是Ubuntu 12.04:
1. 下载Spark
目前最新的版本是spark-0.7.2,下载地址:http://spark-project.org/downloads/
下载后解压到主文件夹,备用。(有点做菜的感觉,呵呵~)
2. 下载Scala
Scala是一种兼顾面向对象和过程的编程语言,Spark需要有Scala的支持才能使用,对于Spark 0.7.2要求的是Scala 2.9.3(在Spark目录下的README文件中写的是2.9.2,估计是没有及时更新),下载地址:http://www.scala-lang.org/downloads
将Scala解压后,将Scala放到/usr/local/shar/目录下,设置系统环境变量PATH,增加SCALA_HOME,我是在~/.profile中设置的,在最后添加:
- #setscalapath
- exportSCALA_HOME=/usr/local/share/scala-2.9.3
- exportPATH=$PATH:$SCALA_HOME/bin
进入在第一步解压的Spark目录,进入conf子目录,执行:
- cpspark-env.sh.templatespark-env.sh
然后修改spark-env.sh内容:设置其中SCALA_HOME=/usr/local/share/scala-2.9.3
3. 安装sbt
所谓sbt,就是Simple Build Tool,是一种用于编译Scala和Java的工具,下载地址:http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html
对于Ubuntu环境,需要下载DEB package,然后双击,输入管理员密码后,就可以自动安装完成。
Spark也支持Maven编译,有兴趣的朋友可以参考:http://spark-project.org/docs/latest/building-with-maven.html
4. 编译Spark
进入解压好的Spark目录下,运行:
初次编译的时间会比较长,在我的环境中最后显示编译时间为1069s。
在编译的过程中,可能会出现一些error,但基本都是关于twitter4j的,可能需要调用一些twitter for Jave的API,不过咱们不用管他,不影响使用。
5. 运行例子程序
在Spark目录下,examples目录中有一些用Java或Scala编写的例子,为了监测编译是否正确,咱们可以试着跑跑。
注意:在跑样例程序之前,需要修改Spark根目录下的run文件,在144行和146行中,必须将“spark-examples-”修改为“spark-examples_”。你可以自己进入examples/target/scala-2.9.3目录,看一下里面jar文件的名称就知道为什么需要这么做了。不然,在跑样例程序的时候,会出现类似下面的错误:
- yn@ubuntu:~/spark-0.7.2$./runspark.examples.SparkPilocal[2]
- 13/06/2109:57:06INFOslf4j.Slf4jEventHandler:Slf4jEventHandlerstarted
- 13/06/2109:57:06INFOspark.SparkEnv:RegisteringBlockManagerMaster
- 13/06/2109:57:06INFOstorage.MemoryStore:MemoryStorestartedwithcapacity323.9MB.
- 13/06/2109:57:06INFOstorage.DiskStore:Createdlocaldirectoryat/tmp/spark-local-20130621095706-c588
- 13/06/2109:57:06INFOnetwork.ConnectionManager:Boundsockettoport33533withid=ConnectionManagerId(ubuntu,33533)
- 13/06/2109:57:06INFOstorage.BlockManagerMaster:TryingtoregisterBlockManager
- 13/06/2109:57:06INFOstorage.BlockManagerMasterActor$BlockManagerInfo:Registeringblockmanagerubuntu:33533with323.9MBRAM
- 13/06/2109:57:06INFOstorage.BlockManagerMaster:RegisteredBlockManager
- 13/06/2109:57:06INFOserver.Server:jetty-7.6.8.v20121106
- 13/06/2109:57:06INFOserver.AbstractConnector:StartedSocketConnector@0.0.0.0:55751
- 13/06/2109:57:06INFObroadcast.HttpBroadcast:Broadcastserverstartedathttp://192.168.1.100:55751
- 13/06/2109:57:06INFOspark.SparkEnv:RegisteringMapOutputTracker
- 13/06/2109:57:06INFOspark.HttpFileServer:HTTPFileserverdirectoryis/tmp/spark-f068aacf-fe52-4db6-bff4-a2aacead476f
- 13/06/2109:57:06INFOserver.Server:jetty-7.6.8.v20121106
- 13/06/2109:57:06INFOserver.AbstractConnector:StartedSocketConnector@0.0.0.0:35958
- 13/06/2109:57:07INFOio.IoWorker:IoWorkerthread'spray-io-worker-0'started
- 13/06/2109:57:07INFOserver.HttpServer:akka://spark/user/BlockManagerHTTPServerstartedon/0.0.0.0:48242
- 13/06/2109:57:07INFOstorage.BlockManagerUI:StartedBlockManagerwebUIathttp://ubuntu:48242
- Exceptioninthread"main"java.lang.NullPointerException
- atjava.net.URI$Parser.parse(URI.java:3019)
- atjava.net.URI.<init>(URI.java:595)
- atspark.SparkContext.addJar(SparkContext.scala:511)
- atspark.SparkContext$$anonfun$2.apply(SparkContext.scala:102)
- atspark.SparkContext$$anonfun$2.apply(SparkContext.scala:102)
- atscala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
- atscala.collection.immutable.List.foreach(List.scala:76)
- atspark.SparkContext.<init>(SparkContext.scala:102)
- atspark.examples.SparkPi$.main(SparkPi.scala:14)
- atspark.examples.SparkPi.main(SparkPi.scala
现在,你可以在Spark根目录下运行:
- ./runspark.examples.SparkPilocal[2]
如果能够看到如下结果,基本就算编译成功了:
- yn@ubuntu:~/spark-0.7.2$./runspark.examples.SparkPilocal[2]
- 13/06/2110:01:19INFOslf4j.Slf4jEventHandler:Slf4jEventHandlerstarted
- 13/06/2110:01:19INFOspark.SparkEnv:RegisteringBlockManagerMaster
- 13/06/2110:01:19INFOstorage.MemoryStore:MemoryStorestartedwithcapacity323.9MB.
- 13/06/2110:01:19INFOstorage.DiskStore:Createdlocaldirectoryat/tmp/spark-local-20130621100119-7b12
- 13/06/2110:01:19INFOnetwork.ConnectionManager:Boundsockettoport60838withid=ConnectionManagerId(ubuntu,60838)
- 13/06/2110:01:19INFOstorage.BlockManagerMaster:TryingtoregisterBlockManager
- 13/06/2110:01:19INFOstorage.BlockManagerMasterActor$BlockManagerInfo:Registeringblockmanagerubuntu:60838with323.9MBRAM
- 13/06/2110:01:19INFOstorage.BlockManagerMaster:RegisteredBlockManager
- 13/06/2110:01:19INFOserver.Server:jetty-7.6.8.v20121106
- 13/06/2110:01:19INFOserver.AbstractConnector:StartedSocketConnector@0.0.0.0:54966
- 13/06/2110:01:19INFObroadcast.HttpBroadcast:Broadcastserverstartedathttp://192.168.1.100:54966
- 13/06/2110:01:19INFOspark.SparkEnv:RegisteringMapOutputTracker
- 13/06/2110:01:19INFOspark.HttpFileServer:HTTPFileserverdirectoryis/tmp/spark-fae55966-53aa-4f14-9dfc-334239949905
- 13/06/2110:01:19INFOserver.Server:jetty-7.6.8.v20121106
- 13/06/2110:01:19INFOserver.AbstractConnector:StartedSocketConnector@0.0.0.0:40454
- 13/06/2110:01:19INFOio.IoWorker:IoWorkerthread'spray-io-worker-0'started
- 13/06/2110:01:19INFOserver.HttpServer:akka://spark/user/BlockManagerHTTPServerstartedon/0.0.0.0:53206
- 13/06/2110:01:19INFOstorage.BlockManagerUI:StartedBlockManagerwebUIathttp://ubuntu:53206
- 13/06/2110:01:19INFOspark.SparkContext:AddedJAR/home/yn/spark-0.7.2/examples/target/scala-2.9.3/spark-examples_2.9.3-0.7.2.jarathttp://192.168.1.100:40454/jars/spark-examples_2.9.3-0.7.2.jarwithtimestamp1371780079811
- 13/06/2110:01:20INFOspark.SparkContext:Startingjob:reduceatSparkPi.scala:22
- 13/06/2110:01:20INFOscheduler.DAGScheduler:Gotjob0(reduceatSparkPi.scala:22)with2outputpartitions(allowLocal=false)
- 13/06/2110:01:20INFOscheduler.DAGScheduler:Finalstage:Stage0(mapatSparkPi.scala:18)
- 13/06/2110:01:20INFOscheduler.DAGScheduler:Parentsoffinalstage:List()
- 13/06/2110:01:20INFOscheduler.DAGScheduler:Missingparents:List()
- 13/06/2110:01:20INFOscheduler.DAGScheduler:SubmittingStage0(MappedRDD[1]atmapatSparkPi.scala:18),whichhasnomissingparents
- 13/06/2110:01:20INFOscheduler.DAGScheduler:Submitting2missingtasksfromStage0(MappedRDD[1]atmapatSparkPi.scala:18)
- 13/06/2110:01:20INFOlocal.LocalScheduler:RunningResultTask(0,0)
- 13/06/2110:01:20INFOlocal.LocalScheduler:RunningResultTask(0,1)
- 13/06/2110:01:20INFOlocal.LocalScheduler:Sizeoftask1is1343bytes
- 13/06/2110:01:20INFOlocal.LocalScheduler:Sizeoftask0is1343bytes
- 13/06/2110:01:20INFOlocal.LocalScheduler:Fetchinghttp://192.168.1.100:40454/jars/spark-examples_2.9.3-0.7.2.jarwithtimestamp1371780079811
- 13/06/2110:01:20INFOspark.Utils:Fetchinghttp://192.168.1.100:40454/jars/spark-examples_2.9.3-0.7.2.jarto/tmp/fetchFileTemp4213276189938108169.tmp
- 13/06/2110:01:20INFOlocal.LocalScheduler:Addingfile:/tmp/spark-cdc3e2f9-e2ea-45ee-99bc-c2cd324f9ee1/spark-examples_2.9.3-0.7.2.jartoclassloader
- 13/06/2110:01:20INFOlocal.LocalScheduler:FinishedResultTask(0,1)
- 13/06/2110:01:20INFOlocal.LocalScheduler:FinishedResultTask(0,0)
- 13/06/2110:01:20INFOscheduler.DAGScheduler:CompletedResultTask(0,1)
- 13/06/2110:01:20INFOscheduler.DAGScheduler:CompletedResultTask(0,0)
- 13/06/2110:01:20INFOscheduler.DAGScheduler:Stage0(mapatSparkPi.scala:18)finishedin0.285s
- 13/06/2110:01:20INFOspark.SparkContext:Jobfinished:reduceatSparkPi.scala:22,took0.326348109s
- Piisroughly3.1431
5. 关于Hadoop版本
Spark是兼容Hadoop HDFS的,所以为了能够正常运行,Spark版本与Hadoop版本必须匹配,对于Spark 0.7.2默认支持的是Hadoop 1.0.4,当需要修改Hadoop版本的时候,可以在根目录下project/SparkBuild.scala中修改HADOOP_VERSION,然后在Spark根目录下重新执行:
或者
关于这两个命令是否功能一样,不是很清楚,待研究,如果有指导的朋友,还请不吝赐教~