ZooKeeper connect/disconnect pattern
Again thank you for your patience.... Is the following pattern normal for a broker that is booting? This is from my zookeeper log. It seems to connect and disconnect multiple times in rapid succession....
View ArticleSeparate broker replication traffic from producer/consumer traffic
Hi all, Is there any way to configure the brokers such that producers & consumers are talking via IP1, while the brokers are replicating between themselves using IP2? I see there are broker...
View Articleproducers limit
Is there a recommended cap for the concurrent producers threads? We plan to have around 4000 connections across cluster writing to kafka, i assume there shouldn't be any performance implications...
View ArticleSome brokers taking a long time to restart
Hello, This is Jie from Pinterest. We did a upgrade from 0.8.0 to 0.8.1 today with rolling restart, but found some brokers took a very long time to restart while some just restarted immediately. Here...
View ArticleWhen is the status of isr updated
Hello, We observed in some cases the isr were outdated: healthy brokers may be missing from isr, or dead brokers may still stay in isr for a while. My guess is the list of isr is only updated upon a...
View ArticleRebalance Logs
Where in the broker logs can I see that a rebalance is happening? Will the state change log tell me this?
View ArticleKafka contrib/hadoop-consumer not working with hadoop 2.2.0
Kafka native hadoop-consumer is compiled with hadoop 0.20 dependencies, so isn't working with hadoop last releases (hadoop 2.2.0) The code hasn't support from two years ago. Is this a discarded option...
View ArticleKicking the tires on 0.8.1...tires kicking back
I am trying to convert a few projects to the latest kafka... Is this the latest artifact? <dependency> <groupId>org.apache.kafka</groupId>...
View ArticleKafka Queue Depth Metrics
Hi, Does anyone know what the JMX metric(s) are for determining the depth of the kafka queue? I basically want to know the amount of messages that are on the queue at a certain point in time. Please...
View ArticleIncreasing the no. of partitions during runtime(programmatically)
Hello, I'm new to Kafka. I was wondering if it's possible to increase the no. of partitions of a topic during runtime(programmatically). I could see that there're command line tools to achieve this. If...
View Articlekafka cluster behind a hardware load balancer
If we have a kafka producer connecting to a Kafka cluster behind a hardware Load Balancer (VIP), will producer be able to send a message to a right partition ? Can one of the brokers in a cluster do a...
View ArticleIs adding paritition disabled in 0.8.1?
Hi team, The following link introduces a script for adding a partition. https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-5.AddPartitionTool However, in the...
View Articlekafka availability
Hi, I perform availability tests on next kafka setup: - 3 nodes(2f+1) for zookeeper(zookeeper1, zookeeper2, zookeeper3) - 2 nodes(f+1) for kafka brokers(id:0,host:kafka.broker1,port:9092;...
View ArticleNullPointerException in broker on notification of ZK session end
I'm seeing a lot of this in my logs on a non-controller broker: 2014-04-02 15:42:23,078] ERROR Error handling event ZkEvent[New session event sent to...
View ArticleFailing broker with errors for Conditional update
Hi Team, I test Kafka 0.8.1 and have problems with missing brokers. What I did: 1. Installed Zookeeper 3.3.6 servers on 3 dedicated machines 2. Installed Kafka 0.8.1 brokers on 3 machines (different...
View ArticleDefaultEventHandler's getPartition's return value
Hi, My apologies - a newbie here. In looking at the getPartition() method, I notice the returned value for keyed message is the "index" of a partition. But when there is no key, then the value...
View ArticleProducer Error: Exception in thread ProducerSendThread
Hi, We are replacing the current data pipeline that sends messages to the tomcat servlet that in turn writes to files and then pushes to HDFS to process. Instead of replacing it entirely at first, we...
View Article0.8.1 high cpu usage
Hi all, To test 0.8.1, I recently upgraded our kafka from 0.8-beta1 to 0.81 on three clusters. On all three of them, I've noticed high cpu usage (20% to more than 70%). Our production is using 0.8-beta...
View ArticleSpring integration?
Hello, My company is looking at Kafka for a backbone to microservices, and we were wondering if anyone had actually done anything with it and Spring Integration (which we are looking at also for...
View ArticleReview for the new consumer APIs
Hi, I'm looking for people to review the new consumers APIs. Patch is posted at https://issues.apache.org/jira/browse/KAFKA-1328 Thanks, Neha
View Article