设为首页 加入收藏

TOP

编译Hadoop-2.4.0之HDFS的64位C++库
2014-11-23 22:10:00 来源: 作者: 【 】 浏览:8
Tags:编译 Hadoop-2.4.0 HDFS 64位

C++库的源代码位于:


这里提供一个直接对这些源文件进行编译的makefile,编译后将打包命名为libhdfs.a. makefile内容为:


CC = gcc
DEFINES = -DG_ARCH_X86_64



CFLAGS += -fPIC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -O3 -D_REENTRANT $(DEFINES)
CXXFLAGS += -pipe -O3 -D_REENTRANT $(DEFINES) -rdynamic


AR = ar cqs
LFLAGS = -rdynamic


OBJECTS = exception.o expect.o hdfs.o jni_helper.o native_mini_dfs.o


TARGET = libhdfs.a


#command, don't change
CHK_DIR_EXISTS= test -d
DEL_FILE = rm -f



first: all
####### Implicit rules


.SUFFIXES: .o .c .cpp .cc .cxx .C .cu


.cpp.o:
$(CXX) -c $(CXXFLAGS) $(INCPATH) -o "$@" "$<"


.cc.o:
$(CXX) -c $(CXXFLAGS) $(INCPATH) -o "$@" "$<"


.cxx.o:
$(CXX) -c $(CXXFLAGS) $(INCPATH) -o "$@" "$<"


.C.o:
$(CXX) -c $(CXXFLAGS) $(INCPATH) -o "$@" "$<"


.c.o:
$(CC) -c $(CFLAGS) $(INCPATH) -o "$@" "$<"

####### Build rules
all: $(AR)


$(AR): $(TARGET)


$(TARGET): $(OBJECTS)
$(AR) $(TARGET) $(OBJECTS)


clean:
-$(DEL_FILE) $(OBJECTS) $(TARGET)


保存好后,直接make. 编译信息如下:


gcc -c -fPIC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -O3 -D_REENTRANT -DG_ARCH_X86_64 -o "exception.o" "exception.c"
gcc -c -fPIC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -O3 -D_REENTRANT -DG_ARCH_X86_64 -o "expect.o" "expect.c"
gcc -c -fPIC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -O3 -D_REENTRANT -DG_ARCH_X86_64 -o "hdfs.o" "hdfs.c"
gcc -c -fPIC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -O3 -D_REENTRANT -DG_ARCH_X86_64 -o "jni_helper.o" "jni_helper.c"
gcc -c -fPIC -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -pipe -O3 -D_REENTRANT -DG_ARCH_X86_64 -o "native_mini_dfs.o" "native_mini_dfs.c"
ar cqs libhdfs.a exception.o expect.o hdfs.o jni_helper.o native_mini_dfs.o


接下来测试一下这个库能不能用。进入以下目录


hadoop-2.4.0-src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test


找到测试源代码,对该文件夹中所有测试代码进行编译.这里再提供一个简单的makefile,内容如下:


LIBS = -L$(JAVA_HOME)/jre/lib/amd64/server/ -ljvm -L../ -lhdfs
INCPATH = -I$(JAVA_HOME)/include -I$(JAVA_HOME)/include/linux -I. -I..
all:
gcc -o hdfs_ops test_libhdfs_ops.c $(INCPATH) $(LIBS)
gcc -o hdfs_read test_libhdfs_read.c $(INCPATH) $(LIBS)
gcc -o hdfs_write test_libhdfs_write.c $(INCPATH) $(LIBS)
gcc -o hdfs_zerocopy test_libhdfs_zerocopy.c $(INCPATH) $(LIBS)


直接make,编译信息如下:


gcc -o hdfs_ops test_libhdfs_ops.c -I/d0/data/lichao/software/java/jdk1.7.0_55/include -I/d0/data/lichao/software/java/jdk1.7.0_55/include/linux -I. -I.. -L/d0/data/lichao/software/java/jdk1.7.0_55/jre/lib/amd64/server/ -ljvm -L../ -lhdfs
gcc -o hdfs_read test_libhdfs_read.c -I/d0/data/lichao/software/java/jdk1.7.0_55/include -I/d0/data/lichao/software/java/jdk1.7.0_55/include/linux -I. -I.. -L/d0/data/lichao/software/java/jdk1.7.0_55/jre/lib/amd64/server/ -ljvm -L../ -lhdfs
gcc -o hdfs_write test_libhdfs_write.c -I/d0/data/lichao/software/java/jdk1.7.0_55/include -I/d0/data/lichao/software/java/jdk1.7.0_55/include/linux -I. -I.. -L/d0/data/lichao/software/java/jdk1.7.0_55/jre/lib/amd64/server/ -ljvm -L../ -lhdfs
gcc -o hdfs_zerocopy test_libhdfs_zerocopy.c -I/d0/data/lichao/software/java/jdk1.7.0_55/include -I/d0/data/lichao/software/java/jdk1.7.0_55/include/linux -I. -I.. -L/d0/data/lichao/software/java/jdk1.7.0_55/jre/lib/amd64/server/ -ljvm -L../ -lhdfs


我们随便生成一个文件,包含1到10这10个数字,并加载到hdfs文件系统。


seq 1 10 > tmpfile
hadoop fs -mkdir /data
hadoop fs -put tmpfile /data
hadoop fs -cat /data/tmpfile
1
2
3
4
5
6
7
8
9
10


ok。现在运行生成的hdfs_read程序,测试一下hdfs的64位C++接口:


./hdfs_read /data/tmpfile 21 32


运行信息如下:


log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
1
2
3
4
5
6
7
8
9
10


--------------------------------------分割线 --------------------------------------


--------------------------------------分割线 --------------------------------------


】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
分享到: 
上一篇对两个奇葩的C语言程序的思考 下一篇在Hadoop中重写FileInputFormat类..

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容: