串联flume

用avro串联

exec-avro-agent.conf 客户端

#agent
exec-avro-agent.sources = exec-source
exec-avro-agent.channels = memory-channel
exec-avro-agent.sinks = avro-sink

#source
exec-avro-agent.sources.exec-source.type = exec
exec-avro-agent.sources.exec-source.command = tail -F /home/hadoop/app/flume-1.6.0-cdh5.7.1/logs/1.log

#channel
exec-avro-agent.channels.memory-channel.type = memory

#sink
exec-avro-agent.sinks.avro-sink.type = avro
exec-avro-agent.sinks.avro-sink.hostname = localhost
exec-avro-agent.sinks.avro-sink.port = 55555


#bind source and sink to channel
exec-avro-agent.sources.exec-source.channels = memory-channel
exec-avro-agent.sinks.avro-sink.channel = memory-channel

启动脚本

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/exec-avro-agent.conf --name exec-avro-agent -Dflume.root.logger=INFO,console

avro-logger-agent.conf 服务端

#agent
avro-logger-agent-1.sources = avro-source
avro-logger-agent-1.channels = memory-channel
avro-logger-agent-1.sinks = logger-sink

#source
avro-logger-agent-1.sources.avro-source.type = avro
avro-logger-agent-1.sources.avro-source.bind = localhost
avro-logger-agent-1.sources.avro-source.port = 55555

#channel
avro-logger-agent-1.channels.memory-channel.type = memory

#sink
avro-logger-agent-1.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-1.sources.avro-source.channels = memory-channel
avro-logger-agent-1.sinks.logger-sink.channel = memory-channel

启动

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent.conf --name avro-logger-agent-1 -Dflume.root.logger=INFO,console

先启动服务端，再启动客户端
往1.log写东西，发现服务端有数据，客户端没数据

负载均衡

客户端

exec-avro-agent-Load-balance.conf

#agent
exec-avro-agent-Load-balance.sources = exec-source
exec-avro-agent-Load-balance.channels = memory-channel
exec-avro-agent-Load-balance.sinks = avro-sink-1 avro-sink-2

#负载平衡
exec-avro-agent-Load-balance.sinkgroups = g1
exec-avro-agent-Load-balance.sinkgroups.g1.sinks = avro-sink-1 avro-sink-2
exec-avro-agent-Load-balance.sinkgroups.g1.processor.type =load_balance
exec-avro-agent-Load-balance.sinkgroups.g1.processor.backoff =true
exec-avro-agent-Load-balance.sinkgroups.g1.processor.selector=round_robin

#source
exec-avro-agent-Load-balance.sources.exec-source.type = exec
exec-avro-agent-Load-balance.sources.exec-source.command = tail -F /home/hadoop/app/flume-1.6.0-cdh5.7.1/logs/1.log

#channel
exec-avro-agent-Load-balance.channels.memory-channel.type = memory

#sink
exec-avro-agent-Load-balance.sinks.avro-sink-1.type = avro
exec-avro-agent-Load-balance.sinks.avro-sink-1.hostname = localhost
exec-avro-agent-Load-balance.sinks.avro-sink-1.port = 55555

exec-avro-agent-Load-balance.sinks.avro-sink-2.type = avro
exec-avro-agent-Load-balance.sinks.avro-sink-2.hostname = localhost
exec-avro-agent-Load-balance.sinks.avro-sink-2.port = 55556

#bind source and sink to channel
exec-avro-agent-Load-balance.sources.exec-source.channels = memory-channel
exec-avro-agent-Load-balance.sinks.avro-sink-1.channel = memory-channel
exec-avro-agent-Load-balance.sinks.avro-sink-2.channel = memory-channel

启动

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/exec-avro-agent-Load-balance.conf --name exec-avro-agent-Load-balance -Dflume.root.logger=INFO,console

两个server端

avro-logger-agent-1.conf

#agent
avro-logger-agent-1.sources = avro-source
avro-logger-agent-1.channels = memory-channel
avro-logger-agent-1.sinks = logger-sink

#source
avro-logger-agent-1.sources.avro-source.type = avro
avro-logger-agent-1.sources.avro-source.bind = localhost
avro-logger-agent-1.sources.avro-source.port = 55555

#channel
avro-logger-agent-1.channels.memory-channel.type = memory

#sink
avro-logger-agent-1.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-1.sources.avro-source.channels = memory-channel
avro-logger-agent-1.sinks.logger-sink.channel = memory-channel

启动

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent-1.conf --name avro-logger-agent-1 -Dflume.root.logger=INFO,console

avro-logger-agent-2.conf

#agent
avro-logger-agent-2.sources = avro-source
avro-logger-agent-2.channels = memory-channel
avro-logger-agent-2.sinks = logger-sink

#source
avro-logger-agent-2.sources.avro-source.type = avro
avro-logger-agent-2.sources.avro-source.bind = localhost
avro-logger-agent-2.sources.avro-source.port = 55556

#channel
avro-logger-agent-2.channels.memory-channel.type = memory

#sink
avro-logger-agent-2.sinks.logger-sink.type = logger

#bind source and sink to channel
avro-logger-agent-2.sources.avro-source.channels = memory-channel
avro-logger-agent-2.sinks.logger-sink.channel = memory-channel

启动

./flume-ng agent --conf /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf --conf-file /home/hadoop/app/flume-1.6.0-cdh5.7.1/conf/avro-logger-agent-2.conf --name avro-logger-agent-2 -Dflume.root.logger=INFO,console

先起两个服务端，再起客户端

然后可以看到两个服务端都可以接受数据，总数为log文件的内容，数据分配的策略，就是我们配置的round_robin算法

容错

这里的10，5是优先级，越大越优先，maxpenalty是冷却时间上限

可参考http://www.aboutyun.com/thread-9170-1-1.html

监控

在server端启动脚本最后面加上

-Dflume.monitoring.type=http -Dflume.monitoring.port=34545

启动然后可以在网页上34545端看到数据