TOP

Windows上搭建Flume运行环境
2019-02-11 14:10:13 】 浏览:279
Tags:Windows 搭建 Flume 运行 环境

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/qq_27384769/article/details/80269346

Windows上搭建Flume运行环境

更多干货

一、概述

1、先安装JDK

2、下载

http://flume.apache.org/FlumeUserGuide.html#kafka-sink

官方下载地址官方用户手册

http://www.apache.org/dyn/closer.lua/flume/1.8.0/apache-flume-1.8.0-bin.tar.gz

二、例子一

根据官方用户手册,创建一个简单例子监听44444端口的输入并在console中输出。

1、进入apache-flume-1.8.0-bin\conf文件夹中创建一个example.conf文件

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

2、使用cmd,进入apache-flume-1.8.0-bin/bin,运行下面命令启动Flume

flume-ng agent --conf ../conf --conf-file ../conf/example.conf --name a1 -property flume.root.logger=INFO,console

在console最后能看到下面这个端口监听提示表示Flume进程正常启动了。

image

3、启动另外一个cmd,使用telnet连接到44444端口并发送信息Hello World!

telnet localhost 44444

4、在Flume的console中可以看到如下提示

image

其他配置

log => flume => kafka

配置一

#agent

producer.sources = s
producer.channels = c
producer.sinks = r

#source
producer.sources.s.type = netcat
producer.sources.s.bind = localhost
producer.sources.s.port = 44444
producer.sources.s.channels = c

producer.sinks.r.type = org.apache.flume.sink.kafka.KafkaSink
producer.sinks.r.kafka.topic = flumeTopic
producer.sinks.r.kafka.bootstrap.servers = localhost:9092
producer.sinks.r.kafka.flumeBatchSize = 20
producer.sinks.r.kafka.producer.acks = 1
producer.sinks.r.kafka.producer.linger.ms = 1
producer.sinks.r.kafka.producer.compression.type = snappy


#Specify the channel for the sink

producer.sinks.r.channel = c
producer.channels.c.type = memory
producer.channels.c.capacity = 1000

配置2

# 定义 agent
a1.sources = src1
a1.channels = ch1
a1.sinks = k1
# 定义 sources
a1.sources.src1.type = exec
a1.sources.src1.command=tail -F /home/centos/log/log
a1.sources.src1.channels=ch1
# 定义 sinks
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = flumeTopic
a1.sinks.k1.brokerList = s201:9092
a1.sinks.k1.batchSize = 20
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.channel = ch1
# 定义 channels
a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000

Windows上搭建Flume运行环境 https://www.cppentry.com/bencandy.php?fid=119&id=207894

】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇自定义Flume的Interceptor,编写.. 下一篇如何在eclipse中开发flume