log retention question
I assume that "log.retention.hours" means the number of hours since the log file was last modified, correct? Or is it since the log file was created? If I set "log.retention.hours" to 48 hours does...
View ArticleUnit for consumer lag
Hi, For kafka 0.8 what are the units of consumer offset,log size and lag reported by the ConsumerOffsetChecker tool ? Thanks,
View ArticleException on Startup. Is it bad or benign.
We have 3 Zookeepers and 3 Kafka Brokers, version 0.8.0. I gracefully shutdown one of the kafka brokers. Question 1: Should I wait some time before starting the broker back up, or can I restart it as...
View ArticleSingle thread, Multiple partitions
Let's say I've a single consumer thread reading off multiple partitions (I'll have around 10K partitions). As per the documentation on...
View ArticleThe easiest way to fetch the topic metadata in my java app
Hi Team, I would like to ask you about the easiest way to fetch topics metadata from my app. My goal is monitoring of the topic size in my java app (when I have also producers). Is there any API in...
View ArticleBuilding Kafka 0.8.1 branch with specific Scala version
Hello Kafka community, I'm trying to import Kafka, 0.8.1 branch, in eclipse IDE. Gradle eclipse plugin is already applied in Kafka build script. If I just run "gradle eclipse", default scala 2.8.0 will...
View ArticleKafka security
What are folk currently doing to secure Kafka and Zk sockets in a cluster ? Firewalls? Ssh tunnels between machines in the cluster and wider servers? Private hand cranked mods to the source code?...
View ArticleUpgrading from 0.8.0 to 0.8.1 one broker at a time issues
Hi, has anyone upgraded their kafka from 0.8.0 to 0.8.1 successfully one broker at a time on a live cluster? I am seeing strange behaviors where many of my kafka topics become unusable (by both...
View Articleconsole producer question
Hey guys. We recently migrated our production cluster from 0.7.2 to 0.8.1. One of the tools in the 0.7 distribution was something called Producer Shell, which we used in some cron jobs to do some...
View ArticleHigh level consumer replaying the messages
Hi , I have set up kafka 0.8 in 3 servers. I have pushed some data into these servers. The number of partitions i use is 12, with a replication factor of 2. We use high level consumer to consume...
View ArticleIssue with kafka cluster with broker not able to send messages.
*Issue : *Kafka cluster goes to an un-responsive state after some time with producers getting Socket time-outs on every request made. *Kafka Version* - 0.8.1 *Machines* : VMs , 2 cores 8gb RAM, linux ,...
View ArticleJust started with Kafka. This test is throwing failed to send message after 3...
I am using the latest version of kafka. package kafka.test; import kafka.consumer.Consumer;import kafka.consumer.ConsumerConfig;import kafka.consumer.ConsumerIterator;import...
View Articleconsumer not consuming messages
Hi, We have 3 node kafka 0.8 setup with zookeepers ensemble. We use high level consumer with auto commit offset false. I am facing some peculiar problem with kafka. When i send some 10-20 messages or...
View ArticleTopic Partitions and Consuming Message at near-real Time
We will be using Kafka for log message transport layer. Log have specific format ( TIMESTAMP HOSTNAME ENVIRONMENT DATACENTER MESSAGE_TYPE APPLICATION_ID PAYLOAD) There are two type of messages...
View ArticleKafka API docs
Hi, The API description at http://kafka.apache.org/documentation.html#api is rather thin -- when you are used to the API docs of other apache projects like hadoop , cassandra , tomcat etc etc. Is there...
View ArticleKafka 0.8.1 protocol description, metadata response issues
Dear Kafka Users, Could you please point me to the protocol description of Kafka 0.8.1? Particularly I would be interested in learning about any metadata request/response changes since 0.8.0. When...
View Articledoubt regarding consumer rebalance exception.
Once i get this exception ERROR consumer.ZookeeperConsumerConnector: [xxxxxxxxxx ], error during syncedRebalance kafka.common.ConsumerRebalanceFailedException: xxxxxxxxx can't rebalance after 4 retries...
View ArticleHadoop Summit Meetups
With the Hadoop Summit in San Jose 6/3 - 6/5 I wondered if any of the LinkedIn geniuses were thinking of putting together a meet-up on any of the associated technologies like Kafka, Samza, Databus,...
View ArticleRe:Hadoop Summit Meetups
great idea. I am interested in attending as well.... From: users [ at ] kafka.apache.org To: users [ at ] kafka.apache.org At: Apr 11 2014 11:40:56 With the Hadoop Summit in San Jose 6/3 - 6/5 I...
View Articlecpu and memory used by kafka broker.
while running kafka broker with 8 topics and 6000 partitions, %cpu usages hover around 300% and virtual memory around 60gb. There are no producers and consumers running. Why is the idle broker taking...
View Article