设为首页 加入收藏

TOP

flume采集数据导入elasticsearch 配置
2018-11-13 16:14:44 】 浏览:347
Tags:flume 采集 数据 导入 elasticsearch 配置

Flume启动一般会报两种错,一种是log4j没有配置,另外一种就是缺少各种jar包。SO:

[root@laiym ~]# cp /usr/local/elasticsearch/lib/*/usr/local/flume/lib/

如果有相同的jar包不用覆盖


下述为flume到elasticsearch的一个配置文件,字段用法详情大家看官方给出的定义。

#文件名称为flume-es.conf

#定义sources,channel和sinks的名称

agent.sources = tail

agent.sinks = elasticsearch

agent.channels = memoryChannel

#配置source的详情

agent.sources.tail.type = exec

agent.sources.tail.command = tail -F /var/log/secure

agent.sources.tail.interceptors=i1 i2 i3

agent.sources.tail.interceptors.i1.type=regex_extractor

agent.sources.tail.interceptors.i1.regex =(\\w+\\s+\\w+\\s+\\d+\\\:\\d+\\\:\\d+)\\s+(\\w+)\\s+(\\w+)

agent.sources.tail.interceptors.i1.serializers = s1 s2s3

agent.sources.tail.interceptors.i1.serializers.s1.name= time

agent.sources.tail.interceptors.i1.serializers.s2.name= hostname

agent.sources.tail.interceptors.i1.serializers.s3.name= service

agent.sources.tail.interceptors.i2.type=org.apache.flume.interceptor.TimestampInterceptor$Builder

agent.sources.tail.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builder

agent.sources.tail.interceptors.i3.hostHeader = host

#配置channel的详情

agent.channels.memoryChannel.type = memory

agentes.channels.channel1.capacity = 1000000

agentes.channels.channel1.transactionCapacity = 5000

#agentes.channels.channel1.keep-alive = 10

#配置sink的详情

agent.sinks.elasticsearch.type=org.apache.flume.sink.elasticsearch.ElasticSearchSink

agent.sinks.elasticsearch.batchSize=100

agent.sinks.elasticsearch.hostNames=127.0.0.1:9300

agent.sinks.elasticsearch.indexName=linux_secure

agent.sinks.elasticsearch.indexType=message

agent.sinks.elasticsearch.clusterName=elasticsearch

agent.sinks.elasticsearch.serializer=org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer

#配置source、sink和channel的详情

agent.sources.tail.channels = memoryChannel

agent.sinks.elasticsearch.channel = memoryChannel


样本日志为linux的secure日志。

Feb 23 17:38:20 laiym sshd[1591]:pam_unix(sshd:session): session closed for user root

Feb 23 17:38:20 laiym sshd[1616]:pam_unix(sshd:session): session closed for user root

Feb 23 17:38:38 laiym sshd[1954]: reverse mappingchecking getaddrinfo for bogon [192.168.141.1] failed - POSSIBLE BREAK-INATTEMPT!

Feb 23 17:38:38 laiym sshd[1954]: Accepted passwordfor root from 192.168.141.1 port 61857 ssh2

Feb 23 17:38:38 laiym sshd[1954]:pam_unix(sshd:session): session opened for user root by (uid=0)

Feb 23 17:50:19 laiym sshd[2019]: reverse mappingchecking getaddrinfo for bogon [192.168.141.1] failed - POSSIBLE BREAK-INATTEMPT!

Feb 23 17:50:19 laiym sshd[2019]: Accepted passwordfor root from 192.168.141.1 port 50289 ssh2

Feb 23 17:50:20 laiym sshd[2019]:pam_unix(sshd:session): session opened for user root by (uid=0)

Feb 24 09:40:51 laiym sshd[1585]:pam_unix(sshd:session): session closed for user root


启动时打开INFO日志和console日志,查看启动状态。

[root@laiym ~]# cd /usr/local/flume/

[root@laiym flume]# ./bin/flume-ng agent -c ./conf/ -f./conf/flume-es.conf -n agent -Dflume.root.logger=INFO,console


在ES中的数据截图:


在kibana中的数据截图:



ok,完美!!!

】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇Flume与Kafka整合完成实时数据处理 下一篇flume自定义Serializer收集日志入..

最新文章

热门文章

Hot 文章

Python

C 语言

C++基础

大数据基础

linux编程基础

C/C++面试题目