Quantcast
Channel: Kafka Timeline
Viewing all articles
Browse latest Browse all 1519

Log retention policy - offset ?

$
0
0
Hi,

We are about to use kafka-0.8 in our production. We have our config to delete messages less than a day i.e 24 hrs. We have our simple consumer which will have its offset in memory and will handle if offsetoutofrange exception occurs.

Suppose if old log messages are getting deleted what will happens to offset ?

Let me give an example,

Suppose we have 10 messages in which 3 messages are older than 24 hrs. My consumer consumed upto message 5 (offset 4). still 5 more msgs yet to consume

After log deletion what will be the offset ?
1. If offset gonna be 0 for message 4, then our consumer will consume only one message. (4 msgs lost ?)
2. If offset gonna be 3 for message 4, then its fine. But what happens if offset keeps on increasing and reaches its maximum ? (just curious)

Correct me if i am wrong.

Thanks in advance

Ranjith Venkatesan

Viewing all articles
Browse latest Browse all 1519

Trending Articles