ConsumerRebalanceFailedException
Hi all, We have a small Kafka cluster (0.7.1 - 3 nodes) in EC2. The load is about 200 million events per day, each being few kilobytes. We have a single node zookeeper. Yesterday suddenly our Kafka...
View ArticleConsumer offset oscillates between 2 values
Hi, I use a Java high level consumer to read from a topic with compression set to snappy. When I query the consumer offset for that group using zookeeper shell, the offset value oscillates between 2...
View ArticleLeader not local for partition error?
Hi all, I'm not sure if I'm doing something wrong or if I missed a step somewhere. A little while ago I successfully got the 0.8 quickstart example to work fine with the console producer/consumer. Then...
View ArticleSample Kafka Producer/Consumer example in Scala
Hello All I have been attempting to build an end-to-end example of producer with multiple consumers using scala. I managed to build a producer, but when I attempt to build a consumer the following...
View ArticleTopic corruption from hardware failure (0.7.1)
Hi everyone, Last week, one of our production Kafka 0.7.1 servers had a hardware failure that resulted in an unclean restart. When the server came back up 5 minutes later, there were two topic...
View ArticleMessage Encoding in Kafka 0.8
I noticed that ProducerData has been removed in 0.8 branch If I'd wish to send a complex message type (encoded message) , how would I do it in 0.8? In 0.7.X (a snippet from Neha's example) val producer...
View ArticleDuplicate Messages on the Consumer
Hello, We recently started seeing duplicate messages appearing at our consumers. Thankfully, the database is set up so that we don't store the dupes, but it is annoying. It's not every message, only...
View ArticleMeetup in Raleigh/Durham, NC
There is a Hadoop meetup happening in Durham next week. I'm presenting an intro to Kafka. I don't suspect there are any Kafka users in the area who are not already members of TriHUG, but sending this...
View ArticleRetrieve most-recent-n messages from kafka topic
We're running Kafka 0.7 and I'm hitting some issues trying to access the newest n messages in a topic (or at least in a broker/partition combo) and wondering if my use case just isn't supported or if...
View ArticleMonitoring with Riemann
Hi, Earlier today I finished work on a library that adds an additional metrics reporter (https://github.com/pingles/kafka-riemann-reporter). I did it because although we were writing the CSV metrics we...
View ArticleFlexible Producers and Consumers
I've read through many sites regarding using Kafka for log aggregation but haven't really found anything that actually talks about how people are shipping their logs into Kafka and consuming them. I'm...
View ArticlePlz help me
I have 2 kafka and 1 zookeeper But When I use kafka producer to send msg to zookeeper ,It just send msg to broker 0,but not broker 1. ./kafka/bin/kafka-producer-perf-test.sh --brokerinfo zk.connect=...
View ArticlePlz help me
I have 2 kafka and 1 zookeeper But When I use kafka producer to send msg to zookeeper ,It just send msg to broker 0,but not broker 1. ./kafka/bin/kafka-producer-perf-test.sh --brokerinfo zk.connect=...
View ArticleReplacing brokers in a cluster (0.8)
I'm planning to upgrade a 0.8 cluster from 2 old nodes, to 3 new ones (better hardware). I'm using a replication factor of 2. I'm thinking the plan should be to spin up the 3 new nodes, and operate as...
View ArticleApache Kafka Question
Hi, I am planning to use Apache Kafka 0.8 to handle millions of messages per day. Now I need to form the environment, like (i) How many Topics to be created? (ii) How many partitions/replications to be...
View ArticleLogo
Hey guys, We need a logo! I got a few designs from a 99 designs contest that I would like to put forward: https://issues.apache.org/jira/browse/KAFKA-982 If anyone else would like to submit a design...
View ArticleMessages TTL setting
Hi, We have a 3 node Kafka cluster. We want to increase the maximum amount of time for which messages are saved in Kafka data logs. Can we change the configuration on one node, stop it and start it and...
View ArticleRecommended log level in prod environment.
The beta release comes with mostly trace level logging. Is this recommended? I notice our cluster produce way too many logs. I set all the level to info currently.
View Articleclean up kafka environmet
Hi All. I have on one machine kafka installation. I needed to move it to another machine and I copied a kafka folder to that machine. when I started kafka in new machine I got such output: [2013-07-23...
View ArticleConsumer stops consuming after 5 gb
After consuming about 5 gb of messages , it stopped consuming and it got stuck at a particular offset. After running the following command I got an error regarding the broker. $ bin/kafka-run-class.sh...
View Article