Zookeeper load burst during replication
Hello, In a multi-broker Kafka 0.8.1.1 setup, I had one broker crashed. I restarted it after some noticeable time, so it started catching up the leader very intensively. During the replication, I see...
View ArticleQuestions about new consumer API
Hi guys, I'm interested in the new Consumer API. http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ I have couple of question. 1. In this doc it says kafka consumer will...
View Articlejvm processes to consume messages
Hi, I have a light load scenerio but I am starting off with kafka because I like how the messages are durable etc. If I have 4-5 topics, am I required to create the same # of consumers? I am assuming...
View ArticleKafkaException: Should not set log end offset on partition
I've installed (for ansible scripting testing purposes) 3 VM's each containing kafka zookeeer clustered together Ubuntu 14.04 Zookeepers are 3.4.6 and kafka 2.11-0.8.2-beta The kafka servers have...
View ArticleHow to create a reliable and fast Kafka Producer in a web server?
Hi, I am using Playframework to build a web server that needs to handle very heavy traffic. And it has a Kafka Producer sending messages to Kafka brokers. Currently I am using the synchronized Kafka...
View ArticleProtocol documentation
Hi I was reading the protocol documentation on the wiki page: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol But it seems that the description is not compete. For...
View ArticleOutOfMemoryException when starting replacement node.
Hi, We were recently trying to replace a broker instance and were getting an OutOfMemoryException when the new node was coming up. The issue happened during the log replication phase. We were able to...
View Articlekafka consumer to write into DB
Hello, all I never developed a kafka consumer, I want to be able to make an advanced kafka consumer in java to consume the data and continuously write the data into postgresql DB. I am thinking to...
View ArticleHow to cleanly shut down ConsumerConnector
Hi, I am using the high-level consumer on 0.8.2-beta. I'm attempting to close a ConsumerConnector (actually a handful of connectors), but am not having much luck actually getting it to close cleanly....
View ArticleBest way to erase consumer state
Hi, What is the best way to erase a stopped, multiprocess consumer's state so that it can be reset (to "smallest offsets", for example) when the consumer is restarted? Deleting...
View ArticleProducer connection unsucessfull
Hi, I've installed the Hortonworks Sandbox and try to get into Kafka. Unfortunately, even the simple tutorial does not work :( http://kafka.apache.org/documentation.html#introduction If I try to send a...
View ArticleWhich consumer to use with Kafka 0.8.2-beta
We are upgrading to 0.8.2 and have rewritten our producer using the new org.apache.kafka.clients.producer.KafkaProducer which works great. Should we also convert the consumers to use...
View Articleunable to create topic
I have running zookeeper 3.4.5-cdh5.2.0 cluster on 3 nodes and managing with cm5.2, Integrated kafka with " http://www.cloudera.com/content/cloudera/en/developers/home/cloudera-labs/apache-kafka.html",...
View ArticleIncreased CPU usage with 0.8.2-beta
Good day, I upgraded a Kafka cluster from v0.8.1.1 to v0.8.2-beta and noticed that the CPU usage on the broker machines went up by roughly 40%, from ~60% to ~100% and am wondering if anyone else has...
View ArticleIn what condition does getoffsetbefore (latest) not return latest offset?
Hi, We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1. In one of our application, we want to get all partitions' latest offsets, so we use getoffsetbefore java API (latest). We believe...
View ArticleHow to Setup MirrorMaker in Generalized way
Hi, I am going to setup Kafka clusters having 3 brokers in Datacenter 1. Topics can be created time to time. Each topic can have varying partitions mostly 1,10 or 20. Each application might have...
View ArticleKafka Issue
Hi, One of the broker seems to have got corrupted in my cluster of 7 brokers. All the topic partitions where this broker was leader are having NoLeader or UnderReplicated partition exceptions. All...
View Articleleaderless topicparts after single node failure: how to repair?
Hi. We've suffered a single node HW failure (broker_id 4) with at least 2 replicas of each topic partition, but some topic parts are now leaderless (all were across 4,5): Topic: topic.with.two.replicas...
View Article