Hello,
I am excited to be trying out Kafka. It sounds like everything I ever wanted in a messaging system, but didn't have.
I am interested in using the High-Level Consumer without losing (consuming) messages that were read from the broker, but not processed by user code (exception thrown, database down, unclean consumer shutdown, whatever). I have read everything I can find on the subject, and the official advice has been to disable auto-commit of the offset, and commit explicitly at a suitable time. This is fair, but not entirely trivial: it applies across all threads using the consumer, and it is not clear what action, if any, should be taken in regards to rebalancing. A good reference discussion can be found at http://mail-archives.apache.org/mod_mbox/kafka-users/201310.mbox/%3CCAA+BczSRjgR=LFfnL+ihsXx6p6sNoVwqjfhrEbULdwwDom+bMQ [ at ] mail.gmail.com%3E
I also understand that there are a few improvements in the works. I already went ahead and voted for them in Jira.
What never seems to be discussed publicly is the fact that there IS a peek() method on ConsumerIterator, and it appears to be just the ticket. Is there anything wrong with doing something like this?
while (it.hasNext()) {
MessageAndMetadata<byte[], byte[]> messageAndMetadata = it.peek();
try {
process(messageAndMetadata);
it.next(); // processed successfully, now consume it
} catch { ... }
In case it matters, I'm dealing with a low message rate, and tracking the latest official release.
Thank you,
Paul
I am excited to be trying out Kafka. It sounds like everything I ever wanted in a messaging system, but didn't have.
I am interested in using the High-Level Consumer without losing (consuming) messages that were read from the broker, but not processed by user code (exception thrown, database down, unclean consumer shutdown, whatever). I have read everything I can find on the subject, and the official advice has been to disable auto-commit of the offset, and commit explicitly at a suitable time. This is fair, but not entirely trivial: it applies across all threads using the consumer, and it is not clear what action, if any, should be taken in regards to rebalancing. A good reference discussion can be found at http://mail-archives.apache.org/mod_mbox/kafka-users/201310.mbox/%3CCAA+BczSRjgR=LFfnL+ihsXx6p6sNoVwqjfhrEbULdwwDom+bMQ [ at ] mail.gmail.com%3E
I also understand that there are a few improvements in the works. I already went ahead and voted for them in Jira.
What never seems to be discussed publicly is the fact that there IS a peek() method on ConsumerIterator, and it appears to be just the ticket. Is there anything wrong with doing something like this?
while (it.hasNext()) {
MessageAndMetadata<byte[], byte[]> messageAndMetadata = it.peek();
try {
process(messageAndMetadata);
it.next(); // processed successfully, now consume it
} catch { ... }
In case it matters, I'm dealing with a low message rate, and tracking the latest official release.
Thank you,
Paul