版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/sinat_33363493/article/details/71124712
1.收集Hive运行日志logs
命令
tail -f /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/logs/hive.log
2.配置文件
# The configuration file needs to define the sources,
# the channels and the sinks.
### define agent
a2.sources = r2
a2.channels = c2
a2.sinks = k2
### define sources
a2.sources.r2.type = exec
a2.sources.r2.command = tail -f /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/logs/hive.log
a2.sources.r2.shell = /bin/bash -c
### define channels
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100
### define sink
a2.sinks.k2.type = hdfs
a2.sinks.k2.hdfs.path = hdfs://hadoop-CDH:8020/user/beifeng/flume/hive-logs/
#a2.sinks.k2.hdfs.path = hdfs://ns1/user/flume/hive-logs/ //当HDFS配置了高可用HA时
a2.sinks.k2.hdfs.fileType = DataStream
a2.sinks.k2.hdfs.writeFormat = Text
a2.sinks.k2.hdfs.batchSize = 10
### bind the soures and sink to the channel
a2.sources.r2.channels = c2
a2.sinks.k2.channel = c2
注意，当HDFS是配置了HA的时候，需要将hadoop的配置文件core-site.xml以及hdfs-site.xml等拷贝到Flume的配置文件夹conf下。
3.运行
bin/flume-ng agent \
-c conf \
-n a2 \
-f conf/flume-tail.conf \
-Dflume.root.logger=DEBUG,console
执行成功
2017-05-03 09:37:54,283 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:261)] Creating hdfs://hadoop-CDH:8020/user/beifeng/flume/hive-logs//FlumeData.1493818674204.tmp
tmp是默认格式，可以改变