Partition key not working properly
Hi all, I'm struggling with how to use the partition key mechanism properly. My logic is set the partition number as 3, then create three partition keys as "0", "1", "2", then use the partition keys to...
View ArticleCan Mirroring Preserve Every Topic's Partition?
Howdy friends, I'd like to mirror the topics on several clusters to a central cluster, and I'm looking at using the default Mirrormaker to do so. I've already done some basic testing on the Mirrormaker...
View ArticlePartition id of consumed message?
Hi all, I am using kafka version 0.8.1.1. In the consumer side, can I find out the partition id of each message I am consuming using high level consumer API? Thanks, Sandeep
View ArticleKafka 0.8.2 release - before Santa Claus?
Hi, People using SPM to monitor Kafka have been anxiously asking us about the 0.8.2 release and we've been telling them December. Is that still the plan? Thanks, Otis
View ArticleKafka 0.8.2 log cleaner
I’m wondering if you could tell me whether log cleaner implemented in 0.8.2 because it seems like it didn’t. Thanks John
View ArticleDoes adding ConsumerTimeoutException make the code more robust?
Hi, I am just wondering if the below snippet ConsumerIterator<byte[], byte[]) it = ... while (True) try { while (it.hasNext()) { ... ... ... } catch (ConsumerTimeoutException e) { // do nothing...
View Articlehow many brokers to set in kafka
Hi, all We are having 3 production server to setup for kafka cluster, I wonder how many brokers to configure for each server. thanks Alec Li
View Articlemaximum number of open file handles
Hi, all I read the comments from http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/, Michael mentioned to increase the maximum number of open file handles for the user kafka to...
View Articlekafka 0.8.1 socket leak? (file descriptor outage)
Hi, We are using kafka 0.8.1 and it's working well for a long time. But there is an issue comes up two days ago: We monitored that kafka is not accepting new messages and get following exception. It's...
View Articlerule to set number of brokers for each server
Dear all I am provision production kafka cluster, which has 3 servers, I am wondering how many brokers I should set for each servers, I set 3 brokers in dev clusters, but I really don't what is the...
View ArticleGetting consumers list from Kafka Metadata API
Hi guys, I was wondering if I could get a list of consumers / consumer groups from Kafka directly without the use of Zookeeper? Bonus point goes if I can get a list of producers too? I'm currently...
View ArticleFailed partition reassignment
Hi, I observed some error messages / exceptions while running partition reassignment on kafka 0.8.1.1 cluster. Being fairly new to this system I'm not sure if these indicate serious failures or...
View ArticlePartition reassignment reversed
I unfortunately do not have any specific logs from these events but I will try and describe the events as accurately as possible to give an idea of the problem I saw. The odd behavior manifested itself...
View ArticleHigh level Consumer API doesnt receive 10MB messages?
Hi all, Consumer doesnt receive message if it is big: When the producer sends 256kb messages to broker, consumer is able to retrieve it, but when producer sends 10MB messages to the broker, the...
View ArticleBest practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1
Hi, We have a kafka cluster that runs Kafka 0.8.1 that we are considering upgrade to 0.8.1.1. The Kafka documentation <http://kafka.apache.org/documentation.html#upgrade> mentions upgrading from...
View ArticleHow to push metrics to graphite - jmxtrans does not work
Hi, I am having a very difficult time trying to report kafka 8 metrics to Graphite. Nothing is listening on 9999 and and no data in graphite. If this method of graphite reporting is know to not work is...
View ArticleRe:How to push metrics to graphite - jmxtrans does not work
hi David, which version do you use kafka? when I use kafka 0.8.0, I write jmxtrans "obj" like this "obj": "\"kafka.server\":type=\"BrokerTopicMetrics\",name=\"AllTopicsBytesOutPerSec\"" . Hope it...
View ArticlePagecache cause OffsetOutOfRangeException
Hi all, I'm using kafka 0.8.0 release now. And I often encounter the problem OffsetOutOfRangeException when cosuming message by simple consumer API. But I'm sure that the consuming offset is smaller...
View ArticleHow to send serialized object in Kafka Producer
Hi, I have written the basic program to send String or byte[] messages to consumer from producer by using java Kafka 0.8.1 . It Works perfectly.But i wanted to send serialized object(Java Bean Object)....
View ArticlePagecache cause OffsetOutOfRangeException
Hi all, I'm using kafka 0.8.0 release now. And I often encounter the problem OffsetOutOfRangeException when cosuming message by simple consumer API. But I'm sure that the consuming offset is smaller...
View Article