设为首页 加入收藏

TOP

flume mac os安装
2019-03-02 02:08:12 】 浏览:316
Tags:flume mac 安装

使用brew安装:

brew install flume

修改配置文件:

vi /usr/local/Cellar/flume/1.6.0/libexec/conf/flume.conf

输入以下内容:

# 指定Agent的组件名称
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# 指定Flume source(要监听的路径)
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /opt/logs/flume
# 指定Flume sink
a1.sinks.k1.type = logger
# 指定Flume channel
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# 绑定source和sink到channel上
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动 :

cd /usr/local/Cellar/flume/1.6.0/libexec/bin/

flume-ng agent –conf conf –conf-file flume.conf –name a1 -Dflume.root.logger=INFO,console

出现以下错:

org.apache.commons.cli.MissingOptionException: Missing required option: n
at org.apache.commons.cli.Parser.checkRequiredOptions(Parser.Java:299)
at org.apache.commons.cli.Parser.parse(Parser.java:231)
at org.apache.commons.cli.Parser.parse(Parser.java:85)
at org.apache.flume.node.Application.main(Application.java:263)
问题原因,少写了–name(应该是双杠号):
flume-ng agent –conf conf –conf-file flume.conf --name a1 -Dflume.root.logger=INFO,console

出现以下错:

2018-12-05 11:06:11,768 ERROR node.Application: A fatal error occurred while running. Exception follows.
java.lang.NullPointerException
at java.io.File.(File.java:277)
at org.apache.flume.node.Application.main(Application.java:299)
问题原因,可能是找不到文件flume.conf(应该是双杠号),同时a1要和flume.conf里的名字对应:
flume-ng agent --conf conf --conf-file …/conf/flume.conf --name a1 -Dflume.root.logger=INFO,console

启动成功:

rockinlius-MacBook-Pro:bin rockinliu$ flume-ng agent --conf conf --conf-file …/conf/flume.conf --name agent1 -Dflume.root.logger=INFO,console
Info: Including Hadoop libraries found via (/usr/local/bin/hadoop) for HDFS access
exec /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp ‘conf:/usr/local/Cellar/flume/1.8.0/libexec/lib/:/usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/common/lib/:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/common/:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/hdfs:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/hdfs/lib/:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/hdfs/:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/mapreduce/lib/:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/mapreduce/:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/yarn:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/yarn/lib/:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/yarn/*’ -Djava.library.path=:/Users/rockinliu/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. org.apache.flume.node.Application --conf-file …/conf/flume.conf --name agent1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/flume/1.8.0/libexec/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2018-12-05 11:18:45,345 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
2018-12-05 11:18:45,349 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:…/conf/flume.conf
2018-12-05 11:18:45,354 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: a1
2018-12-05 11:18:45,354 INFO conf.FlumeConfiguration: Processing:k1
2018-12-05 11:18:45,354 INFO conf.FlumeConfiguration: Processing:k1
2018-12-05 11:18:45,368 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [a1]
2018-12-05 11:18:45,369 WARN node.AbstractConfigurationProvider: No configuration found for this host:agent1
2018-12-05 11:18:45,376 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

日志文件测试:

建一个test.log

输入:

go
to
flume
start
abc
123345
123

查看结果:

2018-12-05 11:53:11,790 INFO avro.ReliableSpoolingFileEventReader: Last read took us just up to a file boundary. Rolling to the next file, if there is one.
2018-12-05 11:53:11,790 INFO avro.ReliableSpoolingFileEventReader: Preparing to move file /opt/logs/flume/test.log to /opt/logs/flume/test.log.COMPLETED
2018-12-05 11:53:15,895 INFO sink.LoggerSink: Event: { headers:{} body: 67 6F go }
2018-12-05 11:53:15,896 INFO sink.LoggerSink: Event: { headers:{} body: 74 6F to }
2018-12-05 11:53:15,896 INFO sink.LoggerSink: Event: { headers:{} body: 66 6C 75 6D 65 flume }
2018-12-05 11:53:15,896 INFO sink.LoggerSink: Event: { headers:{} body: 73 74 61 72 74 start }
2018-12-05 11:53:15,897 INFO sink.LoggerSink: Event: { headers:{} body: 61 62 63 abc }
2018-12-05 11:53:15,897 INFO sink.LoggerSink: Event: { headers:{} body: 31 32 33 33 34 35 123345 }
2018-12-05 11:53:15,897 INFO sink.LoggerSink: Event: { headers:{} body: 31 32 33 123 }

】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇flume命令详解 下一篇nginx ---->flume ----->ka..

最新文章

热门文章

Hot 文章

Python

C 语言

C++基础

大数据基础

linux编程基础

C/C++面试题目