ActiveControllerCount is always will be either 0 or 1 in 3 nodes kafka cluster?
Hi, there is one Mbean in jconsole with name "kafka.controller":type="KafkaController",name="ActiveControllerCount", the Value of this Mbean is always 1 on one broker and 0 on all others remaining...
View ArticleProducer reaches a max of 7Mbps
Hi, I am using kafka producer and broker for a production setup. The expected producer output is 20MBps but I am only getting max of 8MBps. I have verified that we are losing packets by directly...
View ArticleHow to get monitoring stats
Hi, On the Kafka cluster I would like like to get monitoring stats. I do not know java per this page. Monitoring Our monitoring is done though a centralized monitoring system custom to LinkedIn, but it...
View ArticleKafka List Topic Command O/P and Jconsole o/p differs
Hi, I am getting a very surprising case : I am having a 2 nodes Kafka cluster and created some topics. When I am running the command : *bin/kafka-list-topic.sh --zookeeper 192.168.145.119:2181...
View Articleprocessing a batch of messages in a "transaction"
Hi, I have an application which reads messages from a kafka queue, builds up a batch of messages, and then performs some action on that batch. So far I have just used the ConsumerGroup api. However, I...
View ArticleConsumer not consuming for 1st time
Hello I am running a kafka 0.8 consumer with the following configuration : - fetch.size=1000000000 zookeeper.session.timeout.ms=60000 auto.offset.reset=smallest zookeeper.sync.time.ms=200...
View Articleretention size
Hi team, We have 3 brokers in a cluster. The replication factor is 2. I set the default retention size to 3G bytes. I published 12G data to a topic, which is enough to fully load all partitions. I...
View Articleare kafka consumer apps guaranteed to see msgs at least once?
sorry to keep bugging the list, but I feel like I am either missing something important, or I'm finding something wrong w/ the standard consumer api, (or maybe just the docs need some clarification). I...
View Articleany roadmap for releases?
Hi. There is any plan for a beta release of 0.8.1? We have special interest in this improvement https://issues.apache.org/jira/browse/KAFKA-554 Thanks,
View ArticleAdding Kafka dependencies in Play Framework
Let me start off by saying I am a newbie to Kafka, Play as well as Scala. Trying to write an application under Play Framework in which an 'Actor' will write a message to Kafka queue. There's sample...
View ArticleCross version kafka wire-format compatibility
This document describes the wire-format used by kafka at some point during 0.7.x. Is this still valid for 0.8.x? https://cwiki.apache.org/confluence/display/KAFKA/Writing+a+Driver+for+Kafka If not, is...
View ArticleIncreasing number of partitions in kafka-0.8.0-beta1
Hi all, We have kafka-0.8.0-beta1 version installed in our production. Here is what the comment for num.partitions in config file says: # The number of logical partitions *per topic per server.* More...
View ArticleRelationship between socket timeout & consumer wait timeout
I'm trying to use the SimpleConsumer to consume a queue, using long polling. I'm encountering some difficulties when using very long poll times (60s or more). It seems to be conflicting with the socket...
View Article0.72 Kafka - ZK over VPN
I want to use a ZK cluster for my Kafka cluster, which is only available over a cross-country VPN tunnel. The VPN tunnel is prone to resets, every other day or so, perhaps down for a couple of minutes...
View ArticleConsuming from a replica
Hi, I've been looking at the SimpleConsumer example, and that I've noticed that it always reads from the leader, and reacts to leader changes by reconnecting to the new leader. Is it possible to read...
View ArticleKafka Cluster Failover
Hi all! Wikimedia is close to using Kafka to collect webrequest access logs from multiple data centers. I know that MirrorMaker is the recommended way to do cross-DC Kafka, but this is a lot of...
View ArticleLatest Version
I'm running the Kafka 0.8 version downloaded from the downloads page. I'm getting lots of issues with socket timeouts from producer and consumer. I'm also getting errors where brokers that are shut...
View ArticleSegmentList remove CAS in 0.8
I saw the source code SegmentList.scala in 0.8 remove the CAS statement and add lock statement in Log.scala. Does it necessary to use AtomicReference List or can just use @volatile in place? Regards...
View Articlekafka producer - retry messages
Hi. Anyone has build a retry system (durable) for the producer in case the kakfa cluster is down? We have certain messages that must be sent because there are after a transaction that cannot be undo....
View ArticleBelgium, 6 month contract
Good afternoon Apache Kafka users, Let me start off by apologising for taking this route to contact you all, I am sure the last thing you want is a recruiter spamming you on your forum. On the plus...
View Article