设为首页 加入收藏

TOP

flume报错 Configured capacity is 100000000 but the checkpoint file capacity is 1000000
2018-11-13 16:00:16 】 浏览:51
Tags:flume 报错 Configured capacity 100000000 but the checkpoint file 1000000

最近flume运行不是很稳定,本次由于hadoop不能写入,导致flume报错,Configured capacity is 100000000 but the checkpoint file capacity is 1000000,重启flume后问题仍然存在。

1,详细报错如下:

22 Jan 2013 11:07:42,568 INFO [pool-7-thread-1] (org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.channelClosed:209) - Connection to /10.4.203.176:60322 disconnected.

22 Jan 2013 11:07:44,617 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
java.lang.IllegalStateException: Channel closed [channel=file_chan_1]. Due to java.lang.IllegalStateException: Configured capacity is 100000000 but the checkpoint file capacity is 1000000. See FileChannel documentation on how to change a channels capacity.
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:321)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:385)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.IllegalStateException: Configured capacity is 100000000 but the checkpoint file capacity is 1000000. See FileChannel documentation on how to change a channels capacity.
at org.apache.flume.channel.file.EventQueueBackingStoreFile.<init>(EventQueueBackingStoreFile.java:80)
at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:42)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:67)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:36)
at org.apache.flume.channel.file.Log.replay(Log.java:339)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:271)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
... 1 more

2,故障原因:
FileChannel使用了固定大小的checkpoint file,修改channel的能力,简单的方法如下
1,关闭agent
2, 删除或者备份checkpoint目录
3,重启flume agent(如果channel中有大量文件,会导致全面的延迟,建议先关闭数据源,等把file channel中的数据全部写入sink后,等上2分钟,等数据文件删除后,再重启channel
具体详见参考资料这段话,这段意思未完全读明白
,通过解决方法靠谱

The FileChannel actually uses a fixed size checkpoint file -- so it is not possible to set
it to unlimited size (the checkpoint file is mmap-ed to a fixed size buffer). To change the
capacity of the channel, the easiest way off the top of my head is:

* Shutdown the agent.
* Delete all files in the file channel's checkpoint directory. (not the data directories.
Also you might want to move them out, rather than delete to be safe)
* Change your configuration to increase the capacity of the channel.
* Restart the agent - this will cause full replay, so the agent might take sometime to start
up if there are a lot of events in the channel (to avoid this - shutdown the source before
shutting the agent down - so the sink can drain out the channel completely, wait for about
1-2 mins after the channel is empty so that the data files get deleted (this happens only
immediately after a checkpoint - you can verify this by making sure each data dir has only
2 files each), since all events have been sent out - so during restart the channel will be
quite empty, with very little to replay).


参考资料
http://mail-archives.apache.org/mod_mbox/flume-user/201211.mbox/%3CDA66E7657BFBD949829F3C98D73A54CF18E23C@RHV-EXRDA-S11.corp.ebay.com%3E
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
上一篇Apache中组件的日志文件重建 下一篇黑猴子的家:Flume案例 监控文件

最新文章

热门文章

Hot 文章

Python

C 语言

C++基础

大数据基础

linux编程基础

C/C++面试题目