Topic config reset on alter
I am using kafka 8.1.1 If you run an alter on a topic to increase the number of partitions the existing configs are removed from the topic. You can confirm this by doing: bin/kafka-topics.sh —zookeeper...
View ArticleHow to raise a question in forum
HI, I need some help related to Kafka Consumer Lag monitoring.. Kindly let me know how can i post my query to the forum. Regards, Nitin Kumar Sharma.
View ArticleHow to Ingest data into kafka
Hi kafkars, I want to write a java code to ingest csv files into kafka,any help. Thanks, Kishore.
View Articlehow to achieve availability with no data loss when replica = 1?
Hi, in some topics of our system, the data volumn is so huge that we think doing extra replica is a waste of disk and network resource( plus the data is not so important). firstly, we use 1 replica +...
View ArticleReuse ConsumerConnector
Hey Guys, I have a user case that my thread reads from different kafka topic periodically through a timer. The way I am reading from kafka in the timer callback is the following: try { Map<String,...
View ArticleNew Go Kafka Client
Hi, we open sourced a new Go Kafka Client http://github.com/stealthly/go_kafka_client some more info on blog post http://allthingshadoop.com/2014/12/10/making-big-data-go/ for those working with Go or...
View ArticleJMX mbean of Gauge is a java.lang.Object
Kafka 0.8.2-beta consumer, 0.8.1 broker. In our consumer, we retrieve the mbeans "kafka.server":name="*-ConsumerLag",type="FetcherLagMetrics" in order to send partition lag data to our monitoring...
View ArticleProducerData jar file
Hi Experts, kafka.javaapi.producer.ProducerData class is available in 0.8.1 ? I am unable to use it, I copied all the jars in /build/dependant-libs-2.10.1 and libs folders, any help ?
View ArticleVery slow producer
Hi, I’m writing my own producer to read from text files, and send line by line to Kafka cluster. I notice that the producer is extremely slow. It's currently sending at ~57KB/node/s. This is like...
View ArticleHow do I create a consumer group
We're using 0.82 beta and a homegrown c++ async library based on boost asio that has support for the offset api. (apikeys OffsetCommitRequest = 8, OffsetFetchRequest = 9, ConsumerMetadataRequest = 10)...
View ArticleGiven brokers, is it able to know all the zookeepers that brokers connect to
Hi Guys, If I know the brokers. Is there a way to know the zookeeper host from broker list? Thanks! Siyuan
View ArticleClient Offset Storage
My team is using Kafka 0.8.1 and we may not be able to upgrade to 0.8.2 to take advantage of the broker-side commit of client offsets. Is anyone aware of a Java/Scala library for client offset storage...
View Articlekafka async producer takes a lot of cpu
I am using the kafka java api async client (the one that wraps the Scala client). It's dropping quite a bit of data due to the queue being full, and the process ends up taking a lot of cpu too. I am...
View Articlecsv files to kafka
HI Experts, I want to load csv files to kafka, anybody help me to write javacode for this?
View Articlemetrics about how behind a replica is?
Hello, If I understand it correctly, when the number of messages a replica is behind from the leader is < replica.lag.max.messages, the replica is considered in sync with the master and are eligible...
View Articlespark kafka batch integration
hello all, we at tresata wrote a library to provide for batch integration between spark and kafka. it supports: * distributed write of rdd to kafa * distributed read of rdd from kafka our main use...
View ArticlegetOffsetsBefore and latest
i read in several places that getOffsetsBefore does not necessary returns the last offset before the timestamp, because it is basically file based (so it works at the granularity of the files kafka...
View ArticleKafka getting some love from Logstash and Elasticsearch
?Kafka made some headlines with Logstash announcing their latest version beta (1.5) which includes by default a Kafka input and output plugin. Good stuff....
View ArticleKafka design pattern question - multiple user ids
Hello Kafka Experts! Sorry if this has been answered before - I was hoping for a quick response to a naive questions for a newbie like myself! If I have multiple users, how do I split the streams so...
View Article