HighLevelConsumer question
I was under the impression that a KafkaStream would only own a single topic/partition at a time. Is this correct or will it multiple multiple topic-partitions into a single KafkaStream? Thanks, Clark
View ArticleIssue with Upgrade of 0.8.1
Hi, We upgraded to 0.8.1 version of Kafka in TEST,we did load test shutting down 1 broker in the cluster,we are getting below error and cluster becomes unresponsive. Do you guys have any fix for this...
View ArticleConsumerConnector.commitOffsets
Hi, Currently we are using 0.8.1 version of Kafka ?. Does this ConsumerConenctor.commitOffset method commits all partitions for messages in iterator or only the offset of message that is currently...
View ArticleKafka integration...
Are most of you using straight Java or Scala APIs to talk to Kafka, or something else like Spring Integration, Camel, or some other abstraction?
View ArticleGC pauses and rebalance failures
I've got some consumers under decent GC pressure and, as a result, they are having ZK sessions expire and the consumers never recover. I see a number of rebalance failures in the log after the ZK...
View ArticleHigh-Level Consumer and peeking messages
Hello, I am excited to be trying out Kafka. It sounds like everything I ever wanted in a messaging system, but didn't have. I am interested in using the High-Level Consumer without losing (consuming)...
View ArticleKafka upgrade 0.8.0 to 0.8.1 - kafka-preferred-replica-election failure
Hello All, After performing an upgrade of our Kafka 0.8.0 to Kafka 0.8.1, we are receiving a failure in the preferred replica election process. I am not sure if this is a known issue or not. This is a...
View Articlekafka consumer threads are in waiting state and lag is getting increased
Hi. currently we are using *kafka_2.8.0-0.8.0-beta1 and *and high level consumer group to consume messages. Topic has been created with 3 replica and 100 partition so that max 100 threads can consume...
View ArticleController is not being failed over 0.8.1
Hi, We got the following error spamming the logs when broker 1 is the controller and we are shutting it down in controlled manner not kill -9. The leader being switched to broker 2 for all partitions...
View ArticleCluster design distribution and JBOD vs RAID
I am wondering what others are doing in terms of cluster separation. (if at all) For example let’s say I need 24 nodes to support a given workload. What are the tradeoffs between a single 24 node...
View Articleenhancements to kafka-websocket
i've been a busy little bee this week so kafka-websocket now supports keyed messages and user-defined transforms for messages. the latter is pretty raw, improvement suggestions in the form of pull...
View Articlejruby-kafka and logstash-kafka nearly feature complete
Hi everyone, I have updated my jruby-kafka library to support pretty much all the default consumer and producer classes to support all the options included in the docs. Check it out, give some...
View Articleleaders being kicked out of ISR
Hey all, This is xinyao from dropbox and I am trying to push kafka to widely used inside dropbox. I setup up five nodes with 2x replication. The system is running well to handle all the traffic. The...
View ArticleHow to calculate number of partitions?
Hi, I am sizing Kafka setup for the first time. One question I have is, how to determine number of partitions my topic will need. Are there any considerations to be taken in order to determine this...
View ArticleUnable to get off the ground following the "quick start" section
I have tried to use kafka both building form source via gradle as well as un tgz-ing the tarball. Either way, none of the scripts work in the way described in the Quick Start section. Following is an...
View Articlezkclient version
Hi I am trying kafka 0.8.1, and use zkclient version: <dependency> <groupId>com.101tec</groupId> <artifactId>zkclient</artifactId> <version>0.3</version>...
View ArticleReaction to Reactive Streams?
Announcement from Typesafe et al: http://www.reactive-streams.org/ https://typesafe.com/blog/typesafe-announces-akka-streams
View Articletoo many rebalances in my consumer log and fetcher threads are getting stopped
Hi, I am seeing consumer re-balances very frequently and getting socket reconnect exception. log is given below for more insights [2014-04-18...
View ArticleKAFKA-717
I'm trying to apply the patch from KAFKA-717 for 0.8.0-BETA candidate and it fails. Error: Patch failed:project/Build.scala Project/Build.scala patch does not apply. Please let me know if you guys have...
View ArticleReporting Metrics to Apache Kafka and Monitoring with Consumers
Hi, we started a new github project for using Kafka as the central point for all application and infrastructure metrics https://github.com/stealthly/metrics-kafka/. We started off with implementing a...
View Article