In our broker logs, we occasionally see errors like this:
2013-12-23 05:02:08,456 ERROR [kafka-request-handler-2] server.KafkaApis -
[KafkaApi-45] Error when processing fetch request for partition [mytopic,0]
offset 204243601 from consumer with correlation id 130341
kafka.common.OffsetOutOfRangeException: Request for offset 204243601 but we
only have log segments in the range 204343397 to 207423640.
I assume this means there's a consumer that has fallen behind consuming
messages, and the log retention policy has removed messages before they
could be consumed by the consumer.
However, I'm not 100% which consumer it is, and it looks like the only info
we have is the correlation id of the consumer, e.g.:
"from consumer with correlation id 130341"
Is there a way to know which consumer this refers to? It seems there are
far more correlation id's than there are consumers. Would it be possible
to provide a bit more descriptive error message here, so we can immediately
know which consumer is falling behind?
We do see a corresponding entry in the consumer logs too:
2013-12-23 05:02:08,797 WARN
[ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45]
consumer.ConsumerFetcherThread -
[ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45],
Current offset 204243601 for partition [mytopic,0] out of range; reset
offset to 204343397
But it would be nice to be able to also use the broker log to quickly find
consumers with issues.
Also, I'm not sure, why is logging the event as an ERROR in the broker, but
a WARN in the consumer?
Jason
2013-12-23 05:02:08,456 ERROR [kafka-request-handler-2] server.KafkaApis -
[KafkaApi-45] Error when processing fetch request for partition [mytopic,0]
offset 204243601 from consumer with correlation id 130341
kafka.common.OffsetOutOfRangeException: Request for offset 204243601 but we
only have log segments in the range 204343397 to 207423640.
I assume this means there's a consumer that has fallen behind consuming
messages, and the log retention policy has removed messages before they
could be consumed by the consumer.
However, I'm not 100% which consumer it is, and it looks like the only info
we have is the correlation id of the consumer, e.g.:
"from consumer with correlation id 130341"
Is there a way to know which consumer this refers to? It seems there are
far more correlation id's than there are consumers. Would it be possible
to provide a bit more descriptive error message here, so we can immediately
know which consumer is falling behind?
We do see a corresponding entry in the consumer logs too:
2013-12-23 05:02:08,797 WARN
[ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45]
consumer.ConsumerFetcherThread -
[ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45],
Current offset 204243601 for partition [mytopic,0] out of range; reset
offset to 204343397
But it would be nice to be able to also use the broker log to quickly find
consumers with issues.
Also, I'm not sure, why is logging the event as an ERROR in the broker, but
a WARN in the consumer?
Jason