第一步在flume的conf目录下创建文件:hdfs.properties
hdfs.properties
#a1表示代理代理名称
# Name the components on this agent
a1.sources=r1
a1.sinks=k1
a1.channels=c1
#配置sourcel 监控目录是否有文件数据生成
# Describe/configure the source
a1.sources.r1.type=spooldir
a1.sources.r1.spoolDir=/opt/class22/flume-1.7.0/dataDir
a1.sources.r1.channels=c1
a1.sources.r1.fileHeader=false
a1.sources.r1.interceptors=il
a1.sources.r1.interceptors.il.type=timestamp
#配置sinkl 将检测到的数据sink到hdfs上
# Describe the sink
a1.sinks.k1.type=hdfs
a1.sinks.k1.hdfs.path=hdfs://bigdata22.ibeifeng.com:8020/flume
a1.sinks.k1.hdfs.fileType=DataStream
a1.sinks.k1.hdfs.writeFormat=Text
a1.sinks.k1.hdfs.retryInterval=600
a1.sinks.k1.channel=c1
a1.sinks.k1.hdfs.filePrefix=%Y-%m-%d
#通道是以内存方式存储
# Use a channel which buffers events in memory
a1.channels.c1.type=memory
a1.channels.c1.capacity=1000
a1.channels.c1.transactionCapacity=100
第二步:把hadoop中share/hadoop/目录下:
六个jar包commons-configuration-1.6.jar,commons-io-2.4.jar,hadoop-auth-2.7.3.jar,hadoop-common-2.7.3.jar,hadoop-hdfs-2.7.3.jar,htrace-core-3.1.0-incubating.jar
导入到flume的lib目录下面
第三步命令启动flume:bin/flume-ng agent --name a1 --conf conf --conf-fileconf/hdfs.properties