commitOffsets by partition 0.8-beta
Hi, We have use case in DISH where we need to stop the consumer when we have issues in proceeding further to database or another back end. We update offset manually for each consumed message. There are...
View ArticleKafkaException: This operation cannot be completed on a complete request
Hi, I’m performing a producing load test on two node kafka cluster built from the last 0.8.1 branch sources. I have topic loadtest with replication factor 2 and 256 partitions. Initially both brokers...
View Articlehigh level consumer shutdown exception
In kafka 0.8.1, when I shut down high level consumer, I found one exception. I think it is expected, right? 14/04/19 21:12:47 INFO consumer.ZookeeperConsumerConnector:...
View ArticleKafka with multiple zookeper
Hi, As per the documentation we can have multiple broker of kafka on multiple node talking to single Zookeeper instance. Is it possible to use multiple instance of zookeeper with multiple instance of...
View ArticleLeader: -1 on Kafka
Hi, At times, some of kafka topics end up showing the leader as -1. After this, the messages dont get added to the topic nor consumed. I tried digging into why the leader turns -1 *(leader: -1)* Is...
View ArticleToo many replicas after partition reassignment
After doing some partition reassignments, I've ended up with some partitions that have both the old and new brokers assigned. The output of kafka-topics.sh --describe looks like this:...
View Articlehow to measure consumer time for x number of messages
Hi, I am looking for a way to measure time for a consumer to consumer x number of messages. Basically I would like to compare how much it takes the consumer to consume messages and compare them against...
View ArticleLog Retention in Kafka
Hi, I wanted to set the message expiry for a message on a kafka topic. Is there anything like this in kafka? I came across a property - *log.retention.hours* and *topic.log.retention.hours* Had some...
View Articleestimating log.retention.bytes
Hi, Please help me understand how one should estimate upper limit for log.retention.bytes in this situation. Let's say kafka cluster has 3 machines (broker per machine) with 15TB disk space per...
View ArticleStorm Kafka spout
Hi, What is the open source Kafka Spout for storm that people are using ? What is the experience with https://github.com/nathanmarz/storm-contrib/tree/master/storm-kafka ? regards
View ArticleKafka/Zookeeper co-location
Are there any thoughts on running Zookeeper on the same physical nodes that run the Kafka broker? So the loss of a node affects quorum and possibly requires electing new leaders at both the ZK and the...
View ArticleCluster expansion and upgrade
Hello, We are currently running a kafka 0.8-beta cluster. We are planning to expand the existing cluster and use 0.8.1 version on the new nodes. Before upgrading the older ones we want the new ones to...
View ArticleDelete Topic - BadVersionException
Just tried my first topic delete today and it looks like something went wrong on the controller. I issued the command on a test topic and shortly after that a describe looked like:...
View ArticleKafka Performance Tuning
Hi, I am working on a POC where I have 1 Zookeeper and 2 Kafka Brokers on my local machine. I am running 8 sets of Kafka consumers and producers running in parallel. Below are my configurations:...
View Articleperformance testing data to share
I have been testing kafka for the past week or so and figured I would share my results so far. I am not sure if the formatting will keep in email but here are the results in a google doc...all 1,100 of...
View ArticleKafka Replication Behavior
Hi All, I have 3 nodes kafka cluster with below configuration: Replication factor: 2 Number of partitions: 6 topic-name: notification As soon as data is ingested into notification topic, 2 nodes goes...
View ArticleCan't find certain mbeans
Greetings, I'm working with Kafka 0.8.1. and setting up monitoring / metrics trending. I've been exploring mbeans on my kafka broker, and unable to find the following based on this documentation:...
View ArticleApache Kafka 0.8.1.1 Released
The Apache Kafka community is pleased to announce the general availability release of Apache Kafka 0.8.1.1. The 0.8.1.1 is a maintenance release with bug fixes for the 0.8.1 release. Changes in this...
View ArticleNew to Kafka: ZooKeeper and Client
I have a few newbie questions. I need to create a Producer that sends messages to Kafka brokers. Does a machine running a Kafka client (Producer) need its own instance of ZooKeeper running? Or does is...
View Article