NotLeaderForPartitionException while doing performance test
Hi, All I am doing performance test by bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test-rep-three 500000000 100 -1 acks=1 bootstrap.servers=10.100.98.100:9092,...
View Articleconnection error among nodes
Hi, Experts Our cluster is a 3 nodes cluster, I simply test producer locally, see bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test-rep-three 1000000 3000 -1 acks=1...
View Articlequestion about jmxtrans to get kafka metrics
Hi, All I installed jmxtrans and graphite, wish to be able to graph stuff from kafka, but firstly I start the jmxtrans and getting such errors, (I use the example graphite json). ./jmxtrans.sh start...
View Articlehow to order message between different partition
Hi all, I have a topic with 3 partitions, and each partion has its sequency in kafka. How to order message between different partion? Anyone has experiences about that, thanks. Thanks & Regards...
View Articlekafka monitoring
Hello, All I understand many of you are using jmxtrans along with graphite/ganglia to pull out metrics, according to https://kafka.apache.org/081/ops.html, it says "The easiest way to see the available...
View ArticleMonitor data to missing topic
Hi all, I'm trying to find an existing metric, or method to monitor for data being sent to a topic that doesn't exist. Is there a JMX stat I should look at now, or some other way you catch this sort of...
View ArticleOffset backup and restore in Kafka v0.8.2
We currently checkpoint partition offsets from ZK into an external database, and we have a process for restoring those offsets to ZK in case we ever need to replay data from Kafka. Since offsets are...
View ArticleProducer exception handling with required.ack = -1
Hi All, Just want to double check with you regarding producers using required.acks = -1. - Producer is guaranteed to receive responses within certain time. Because the satisfied request checking is for...
View Articlezookeeper monitoring
Hi, all I've just figured out the monitoring of kafka by jconsole, I want to do the same thing to zookeeper. Zookeeper site says "The class *org.apache.zookeeper.server.quorum.QuorumPeerMain* will...
View ArticleCreate topic, then fetch offset
(1) Create topic (2) Send a FetchOffsetRequest for a partition in the new topic Results in error code 3, or UnknownTopicOrPartition. How can it be unknown, since I just created it? Shouldn’t it return...
View Articlemetric-kafka problems
Hello, all I like to use the tool metrics-kafka which seems to be attractive to report kafka metric and use graphite to graph metrics, however I am having trouble to make it work. In...
View ArticleKafka broker shutting down after running fine for 1-2 hours
Hello, We have just started using kafka. Our test setup consists of a single partition. We have integrated kafka to our system successfully with the help of sample Producer example and Low-level...
View ArticleUsing Kafka for Event Sourcing
Hi, after having read http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying, I am considering Kafka for an application build...
View Articlenew to kafka
I am fairly new to Kafka and working on building a stream processing system using Kafka and Storm. I am looking for couple of references: 1. Reference architecture for Kafka hardware setup that will...
View ArticleGet replication and partition count of a topic
Hi All, I want to get the replication and partition count of a topic. I tried the following piece of code: java.util.Set<String> topics = new HashSet<String>(); topics.add("topicName");...
View ArticleKafka Cluster Monitoring and Documentation of Internals (JMX Metrics) of...
Hi Kafka Team, I am trying to find out Kafka Internal and how a message can be corrupted or lost at brokers side. I have refer to following documentations for monitoring:...
View ArticleThread safety of Encoder implementations
Hi Kafka Users, I have written my own implementation of the kafka Encoder class for serializing objects to Messages. It uses Kryo, which is a non-thread safe java serialization library. I'm using Kafka...
View ArticleConsumer not getting data when there is a big lag in the topic
Hi, I am been facing some JAVA high level consumer related issues lately and would like to understand more on this. We have 9 bare-metals (48 core, 250 GB, Terabytes of Hard disks) running *Kafka...
View Articlenetwork connection between kafka nodes
Hello, Kafka experts I have a production cluster which has three nodes(.100, .101, .102) I am using a C# producer to publish data to kafka brokers, it works for a while but started to lose connection...
View ArticleHow to handle authentication system wide with kafka as the transport layer?
Hello I've been reading about kafka and its use in a distributed system, authentication handling remains a mystery to me. Indeed, when the user connects to the website, many services are triggered :...
View Article