1.实现功能:
扇入:flume可以实现从数以百计的Web servers中收集信息,然后将日志信息传送到十几个agent服务器,最后写到hdfs上。本文实例采用的就是实现这个扇入功能:
flume1:收集4666端口信息,并输送到Avro Sink
flume2:监控日志信息,并输送到Avro Sink
flume3:收集flume1 Avro Sink和flume2 Avro Sink的数据,写入到hdfs上。
2.代理
(1)flume1
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = bigdata.ibeifeng.com
a1.sources.r1.port = 4666
# Describe the sink
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = bigdata.ibeifeng.com
a1.sinks.k1.port = 6666
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
#channe最多存储1000事件
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
(2)flume2
a2.sources = r2
a2.sinks = k2
a2.channels = c2
#定义source的类型
a2.sources.r2.type = exec
a2.sources.r2.command = tail -f /var/log/httpd/access_log
a2.sources.r2.channels = c2
#定义channel
a2.channels.c2.type = memory
#存储1000个event事件
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100
#定义sink
a2.sinks.k2.type = avro
a2.sinks.k2.hostname = bigdata.ibeifeng.com
a2.sinks.k2.port = 6666
a2.sinks.k2.channel = c2
(3)flume3
a3.sources = r3
a3.sinks = k3
a3.channels = c3
#定义source的类型
a3.sources.r3.type = avro
a3.sources.r3.bind = bigdata.ibeifeng.com
a3.sources.r3.port = 6666
a3.sources.r3.channels = c3
#定义channel
a3.channels.c3.type = memory
#存储1000个event事件
a3.channels.c3.capacity = 1000
a3.channels.c3.transactionCapacity = 100
#定义sink
a3.sinks.k3.type = hdfs
a3.sinks.k3.hdfs.path = hdfs://bigdata.ibeifeng.com:8020/opo
a3.sinks.k3.channel = c3
3.启动
bin/flume-ng agent --name a3 --conf conf --conf-file conf/avro-hdfs.properties &
bin/flume-ng agent --name a2 --conf conf --conf-file conf/apache-avro.properties &
bin/flume-ng agent --name a1 --conf conf --conf-file conf/flume-test2.properties &
(经测试,成功!)