设为首页 加入收藏

TOP

Hadoop pipes初学笔记
2019-05-16 00:23:12 】 浏览:74
Tags:Hadoop pipes 初学 笔记
版权声明:IT资讯科技 https://blog.csdn.net/qq_38460284/article/details/90232118

1.安装配置好Hadoop
常用命令:
hadoop dfs -ls path
hadoop dfs -rmr file
hadoop dfs -mkdir path
hadoop dfs -cat file
2.找个wordcount程序,命名为wordcount.cpp
可以是:http://wiki.apache.org/hadoop/C++WordCount
也可以是hadoop安装路径下的:/usr/local/hadoop-0.20.2/src/examples/pipes/impl/wordcount-simple.cc
3.写Makefile
HADOOP_INSTALL=/usr/local/hadoop-0.20.2
PLATFORM=Linux-i386-32

CC = g++
CPPFLAGS = -m32 -I$(HADOOP_INSTALL)/c++/$(PLATFORM)/include

wordcount: wordcount.cpp
$(CC) $(CPPFLAGS) $< -Wall -L$(HADOOP_INSTALL)/c++/$(PLATFORM)/lib -lhadooppipes -lhadooputils -lpthread -g -O2 -o $@
###
cat /proc/cpuinfo 查看cpu是intel的还是amd的,对应修改PLATFORM。
4.执行:
上传wordcount.cpp文件作为输入文件:hadoop fs -put wordcount.cpp input.txt
上传可执行文件: hadoop fs -put wordcount bin/wordcount
运行代码:
hadoop pipes \
-D hadoop.pipes.java.recordreader=true \
-D hadoop.pipes.java.recordwriter=true \
-input input.txt \
-output output \
-program bin/wordcount
查看结果:
hadoop dfs -cat output/*

推荐阅读文章


mp;mid=100000042&idx=5&sn=6fee18778be5646423278ab174fd1f46&chksm=4f68fe0a781f771c093f541c875dae43183fe49c1eb3a482a06dc0f7aaccc241309863c5fee4&xtrack=1&scene=0&subscene=90&sessionid=1557816542&clicktime=1557816723&ascene=7&devicetype=android-26&version=27000481&nettype=ctnet&abtest_cookie=BQABAAgACgALABIAEwAGAJ%2BGHgBWmR4Aw5keANyZHgDimR4A85keAAAA&lang=zh_CN&pass_ticket=8W5MIKb%2FtX1WSbZTwoN%2FlGj0D9eK3p%2FWwM3V4hSialiV1TSZj%2FcTL2BgLpbZxEPI&wx_header=1" rel="nofollow">大数据工程师在阿里面试流程是什么?

学习大数据需要具备怎么样基础?

年薪30K的大数据开发工程师的工作经验总结?


编程开发网
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇结合案例讲解MapReduce重要知识点.. 下一篇2018-3-7      ..

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容:

array(4) { ["type"]=> int(8) ["message"]=> string(24) "Undefined variable: jobs" ["file"]=> string(32) "/mnt/wp/cppentry/do/bencandy.php" ["line"]=> int(217) }