Can I delete old log files manually?
My apologies for mail bombing the list. I'm banging my head against a production issue. In short, can I delete old log and index files manually? I have two brokers out of five that are hosting ~1200...
View ArticleZooKeeper connect/disconnect pattern
Again thank you for your patience.... Is the following pattern normal for a broker that is booting? This is from my zookeeper log. It seems to connect and disconnect multiple times in rapid succession....
View ArticleSeparate broker replication traffic from producer/consumer traffic
Hi all, Is there any way to configure the brokers such that producers & consumers are talking via IP1, while the brokers are replicating between themselves using IP2? I see there are broker...
View Articleproducers limit
Is there a recommended cap for the concurrent producers threads? We plan to have around 4000 connections across cluster writing to kafka, i assume there shouldn't be any performance implications...
View ArticleSome brokers taking a long time to restart
Hello, This is Jie from Pinterest. We did a upgrade from 0.8.0 to 0.8.1 today with rolling restart, but found some brokers took a very long time to restart while some just restarted immediately. Here...
View ArticleWhen is the status of isr updated
Hello, We observed in some cases the isr were outdated: healthy brokers may be missing from isr, or dead brokers may still stay in isr for a while. My guess is the list of isr is only updated upon a...
View ArticleRebalance Logs
Where in the broker logs can I see that a rebalance is happening? Will the state change log tell me this?
View ArticleKafka contrib/hadoop-consumer not working with hadoop 2.2.0
Kafka native hadoop-consumer is compiled with hadoop 0.20 dependencies, so isn't working with hadoop last releases (hadoop 2.2.0) The code hasn't support from two years ago. Is this a discarded option...
View Articledata loss on replicated topic
My company currently testing Kafka for throughput and fault tolerance. We've set up a cluster of 5 Kafka brokers and are publishing to a topic with replication factor 3 and 100 partitions. We are...
View ArticleHelp in setting up Kafka cluster
Hi, First time I am trying to setup new kafka cluster. I have tried sudo cluster with cli based kafka producer and consumer. Having difficulties with setting up log aggregation kafka producer and...
View ArticleHow are rebalance failures raised to consumers?
The pattern for creating and operating consumers that we use is to create the consumer connector, create the streams and then consume each stream by waiting on the iterator. If a rebalance occurs and...
View ArticleLog retention policy - offset ?
Hi, We are about to use kafka-0.8 in our production. We have our config to delete messages less than a day i.e 24 hrs. We have our simple consumer which will have its offset in memory and will handle...
View ArticleError in fetch name
We see the following messages in the broker logs whenever we reboot a broker. These messages filled up 200MB of log files in less than 1 minute. Are these normal? For reference we have enabled...
View ArticleZK session kill makes high level consumer replay
Hi While testing kafka 0.8 consumer's zk resilience, I found that on the zk session kill and handleNewSession() is called, high level consumer is replaying messages. Is this know issue? I am attaching...
View ArticlePlease add Perl client on your wiki
Kafka 0.8.x Pure Perl, Consumer and Producer implementations included. Zookeeper integration. GZIP and Snappy compression not supported. Maintainer: Sergey Gladkov, sgladkov [ at ] trackingsoft.com...
View ArticleJava API to monitor Consumer Offset and Lag
Hi, I am trying to write a groovy script to get consumer offsets and lag for our kafka cluster. Is there a Java API to get this information ? Thanks regards,
View ArticleKafka and authentication
Hi All, I was googling around for info on securing kafka. The best document I could find was https://cwiki.apache.org/confluence/display/KAFKA/Security, which is “kind of old”. It is not clear if any...
View Articleassembly dependency jar
Hi team, I remember for kafka 0.8, there is a kafka jar with all the dependencies in it. 0.81 does not seem to have a jar. Can anybody confirm that? Thanks. Libo
View ArticleKeeperException
Whenever a session is expired by ZooKeeper I see the following messages (one per consumer I think) in the ZooKeeper log: 2014-03-25 00:05:12,953 - INFO [ProcessThread:-1:PrepRequestProcessor [ at ]...
View ArticleCan I delete old log files manually?
My apologies for mail bombing the list. I'm banging my head against a production issue. In short, can I delete old log and index files manually? I have two brokers out of five that are hosting ~1200...
View Article