Programmatic Kafka version detection/extraction?
Hi, Is there a way to detect which version of Kafka one is running? Is there an API for that, or a constant with this value, or maybe an MBean or some other way to get to this info? Thanks, Otis
View Articlezookeeper snapshot files eat up disk space
Hi, My zookeeper 'dataLogDir' is eating up my disk with tons of snapshot files. what are these files? what files can I delete? are week old files disposable? This folder only gets bigger... How can I...
View Articlespikes in producer requests/sec
We're seeing periodic spikes in req/sec rates across our nodes. Our cluster is 10 nodes, and the topic has a replication factor of 3. We push around 200k messages / sec into Kafka. The machines are...
View ArticleSecurity in 0.8.2 beta
Hi, Is there a way to secure the topics created in Kafka 0.8.2 beta? The need is to ensure no one is asked to read data from the topic without authorization. Regards Kashyap
View Articlefirst message is lost
Hi, For a non-existent topic, the consumer and producer are set up. Then if the producer sends the first message, producer gets this exception: [2014-11-12 16:24:28,041] WARN Error while fetching...
View ArticleGetting Simple consumer details using MBean
Hi, I want to get the simple consumer details using MBean as described here <https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring>. But these bean names are not...
View Article0.8.2 producer with 0.8.1.1 cluster?
Hi, Is the new producer 0.8.2 supposed to work with 0.8.1.1 cluster? Shlomi
View ArticleOrder of consumed messages in the same partition.
Hi all, I would like to know if there is any possibility to consume one message for a given topic and partition, using the high-level consumer, that has the offset lower than the last consumed offset...
View ArticleKafka in a docker container stops with no errors logged
Hello, We're using Kafka 0.8.1.1 and we're trying to run it in a Docker container. For the most part, this has been fine, however one of the containers has stopped a couple times and when I look at the...
View ArticleBroker keeps rebalancing
Hi there, My kafka client is reading a 3 partition topic from kafka with 3 threads distributed on different machines. I am seeing frequent owner changes on the topics when running:...
View ArticleHow to keep-alive connection from producer to broker?
Hi, I would like to know if there is a way/configuration to keep-alive connections from producer/client to broker? I observed that connection is getting closed every time, which takes considerable...
View ArticleKafka to transport binary files
I'm thinking of using Kafka for transporting binary files (tiff, jpeg, pdf). These files are anywhere between 10 KB to 5MB. Thought behind considering Kafka is - It serves as a staging area for the...
View Articlehow to configure kafka over firewall
Hi, I would like to know the way to configure kafka over firewall. We have firewall where it filters the connections based on protocol and there are security checks in place. Our broker 0.8.1 resides...
View ArticleOne of two consumers is always Idle though I have 2 partitions
Dear Developers, I am 2nd year masters student at IIT. I am using Kafka for one of my research projects.My question is the following: 1. I have a producer, consumer and a broker(that contains 1st...
View Articlelog4j dir?
Hi guys, Just notice kafka.logs.dir in log4j.properties doesn't take effect It's always set to *$base_dir/logs* in kafka-run-class.sh LOG_DIR=$base_dir/logs KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=$LOG_DIR...
View ArticlePlugable metadata store
Hello Apache Kafka community, Is it already possible to configure/use a different metadata store (topics, consumer groups, consumer to partition assignments, etc.) instead of ZooKeeper? If not, are...
View ArticleKafka broker cannot start after running out of disk space
Hello, I've run into an issue with Kafka 0.8.1.1 broker. The broker stopped working after the disk it was writing to ran out of space. I freed up some space and tried to restart the broker. It started...
View ArticleConsumer reading from same topic twice without commiting
Hi, I am trying to understand the behavior of Kafka Consumer in Java. Consider a scenario where a consumer is reading from a partition with auto.comiit disabled and only one partition in the topic. The...
View Article