最近也是有人问我kafka的auto.offset.reset设置为earliest后怎么结果和自己想象的不一样呢,相信很多人都对这个参数心存疑惑,今天来详细讲解一下:
kafka-0.10.1.X版本之前:auto.offset.reset 的值为smallest,和,largest.(offest保存在zk中)
kafka-0.10.1.X版本之后:auto.offset.reset 的值更改为:earliest,latest,和none (offest保存在kafka的一个特殊的topic名为:__consumer_offsets里面)
顾名思义,earliest就是从最开始消费数据,latest即为从最新的数据开始消费,但我们在使用的时候发现并不是这样的.下面就来详细测试一下.
先看一下官网对auto.offset.reset的解释吧(测试版本为0.10.2.1)
这个解释也比较的抽象,不太好理解,所以我们还是自己动手去验证一下吧.
新建一个topic,名为jason_1122,先查看一下这个topic的信息:
Topic:jason_1122 PartitionCount:10 ReplicationFactor:3 Configs:
Topic: jason_1122 Partition: 0 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
Topic: jason_1122 Partition: 1 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
Topic: jason_1122 Partition: 2 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
Topic: jason_1122 Partition: 3 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
Topic: jason_1122 Partition: 4 Leader: 1 Replicas: 1,2,3 Isr: 1,2,3
Topic: jason_1122 Partition: 5 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
Topic: jason_1122 Partition: 6 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
Topic: jason_1122 Partition: 7 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2
Topic: jason_1122 Partition: 8 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3
Topic: jason_1122 Partition: 9 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
可以看到jason_1122这个topic有10分分区,3分副本.
测试的过程:
1.先写入10条数据
2.设置offest为false,分别测试earliest,latest,和none,这三种情况
3,设置offest为true,分别测试earliest,latest,和none,这三种情况
3.在写入10条数据
4.设置offest为false,分别测试earliest,latest,和none,这三种情况
5.设置offest为true,分别测试earliest,latest,和none,这三种情况
测试一:
先写入10条数据, 设置自动提交offest为false,然后分别测试auto.offset.reset设置为earliest,latest,和none的情况.
测试结果:
(1).当auto.offset.reset为earliest时,消费到了10条数据如下:
0--null--hello jason what are you doing--1542886948685--6
0--null--hello jason what are you doing--1542886948685--3
0--null--hello jason what are you doing--1542886948686--0
0--null--hello jason what are you doing--1542886948658--8
0--null--hello jason what are you doing--1542886948684--7
0--null--hello jason what are you doing--1542886948683--5
0--null--hello jason what are you doing--1542886948684--4
0--null--hello jason what are you doing--1542886948683--2
0--null--hello jason what are you doing--1542886948685--1
(2).当auto.offset.reset为latest时,没有消费到数据.
(3).当auto.offset.reset为none时,抛出异常如下:
Exception in thread "main" org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined offset with no reset policy for partition: jason_1122-4
at org.apache.kafka.clients.consumer.internals.Fetcher.resetOffset(Fetcher.java:369)
at org.apache.kafka.clients.consumer.internals.Fetcher.updateFetchPositions(Fetcher.java:247)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:1602)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1035)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
at kafka.KafkaConsumer$.main(KafkaConsumer.scala:19)
at kafka.KafkaConsumer.main(KafkaConsumer.scala)
测试二:
设置自动提交offest为true,然后分别测试auto.offset.reset设置为earliest,latest,和none的情况.
测试结果:
(1)当auto.offset.reset为earliest时,消费到了10条数据如下:
0--null--hello jason what are you doing--1542886948658--8
0--null--hello jason what are you doing--1542886948684--7
0--null--hello jason what are you doing--1542886948685--6
0--null--hello jason what are you doing--1542886948683--5
0--null--hello jason what are you doing--1542886948684--4
0--null--hello jason what are you doing--1542886948685--3
0--null--hello jason what are you doing--1542886948683--2
0--null--hello jason what are you doing--1542886948685--1
0--null--hello jason what are you doing--1542886948686--0
(2)当auto.offset.reset为latest时,没有消费到数据.
(3)当auto.offset.reset为none时,没有消费到数据,也没有抛出异常.
测试三:
在写入10条数据,一共为20条数据,设置自动提交offest为false,然后分别测试auto.offset.reset设置为earliest,latest,和none的情况.
测试结果:
(1)当auto.offset.reset为earliest时,消费到了10条数据如下:
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1
0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
(2)当auto.offset.reset为latest时,同样消费到了10条数据如下:
0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1
(3)当auto.offset.reset为none时,同样消费到了10条数据如下:
0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1
测试四:
设置自动提交offest为true,然后分别测试auto.offset.reset设置为earliest,latest,和none的情况.
测试结果:
(1)当auto.offset.reset为earliest时,消费到了10条数据如下:
0--null--hello jason what are you doing--1542888276631--9
1--null--hello jason what are you doing--1542888276631--6
1--null--hello jason what are you doing--1542888276631--3
1--null--hello jason what are you doing--1542888276632--0
1--null--hello jason what are you doing--1542888276632--8
1--null--hello jason what are you doing--1542888276631--7
1--null--hello jason what are you doing--1542888276631--5
1--null--hello jason what are you doing--1542888276631--4
1--null--hello jason what are you doing--1542888276607--2
1--null--hello jason what are you doing--1542888276631--1
(2)当auto.offset.reset为latest时,没有消费到数据
(3)当auto.offset.reset为none时,也没有消费到数据
根据上面的测试,得到最终的结论:
如果存在已经提交的offest时,不管设置为earliest 或者latest 都会从已经提交的offest处开始消费
如果不存在已经提交的offest时,earliest 表示从头开始消费,latest 表示从最新的数据消费,也就是新产生的数据.
none topic各分区都存在已提交的offset时，从提交的offest处开始消费；只要有一个分区不存在已提交的offset，则抛出异常
如果有写的不对的地方,欢迎大家指正,如果有什么疑问,可以加QQ群:340297350,谢谢
