设为首页 加入收藏

TOP

spark Shell 启动和提交任务
2019-04-14 13:15:55 】 浏览:27
Tags:spark Shell 启动 提交 任务
#cd training/spark/bin
#./spark-shell --master spark://192.168.56.21:7077    // 单机版

#./spark-shell --master spark://192.168.56.2121:7077 --executor-memory 512m --total-executor-cores 2
scala> sc.parallelize(List("hello scala","hello java","hello scala")).flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
res0: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[3] at reduceByKey at <console>:25

scala> sc.parallelize(List("hello scala","hello java","hello scala")).flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).collect
res1: Array[(String, Int)] = Array((scala,2), (hello,3), (java,1))              

scala> sc.parallelize(List("hello scala","hello java","hello scala")).flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).sortBy(_._2).collect
res2: Array[(String, Int)] = Array((java,1), (scala,2), (hello,3))




编程开发网
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇Scala   WordCount 下一篇2018-11-12Pyspark win环境配置参..

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容:

array(4) { ["type"]=> int(8) ["message"]=> string(24) "Undefined variable: jobs" ["file"]=> string(32) "/mnt/wp/cppentry/do/bencandy.php" ["line"]=> int(217) }