Using MirrorMaker to move some data between clusters
Hi all, I have just a quick question. I was wondering if MirrorMaker is the right solution for this task, basically I need to move a subset of the data from a production cluster to a test cluster. My...
View ArticleKafka AWS deployment + UI console
We are going to deploy Kafka in Production and also monitor it via console. (e.g. State of partitions in Broker- leader and slaves, state of consumers ) Is there out-of-the-box solution? What is the...
View Articleconnect.timeout.ms porperty not present in kafka 8 producer config
Kafka 07 has following property for producer. connect.timeout.ms5000the maximum time spent bykafka.producer.SyncProducer trying to connect to the kafka broker. Once it elapses, the producer throws an...
View ArticleProducer connection timing out
Even though I am able to ping to the broker machine from my producer machine , the producer is throwing below expcetion while connecting to broker. I wanted to increase time out for producer but...
View ArticleLoad Balancing Consumers or Multiple consumers reading off same topic
Hi, I'm not even sure if this is a valid use-case, but I really wanted to run it by you guys. How do I load balance my consumers? For example, if my consumer machine is under load, I'd like to spin up...
View ArticleNot able to create topic
Hi, I am trying to create a topic using TopicCommand and I get an error back. I see lot of rebalance attempt logs. Can someone help me with this issue? 12702927406-17ccd847], begin rebalancing consumer...
View ArticleConnections from kafka consumer
I have few questions regarding Kafka Consumer. In kafka properties we only mention the zookeeper ip's to connect to. But I assume the consumer also connects to Kafka broker for actually consuming the...
View Articlepostgresql consumer
Hi, All I setup a kafka cluster, and plan to publish the messages from Web to kafka, the messages are in the form of json, I want to implement a consumer to write the message I consumer to postgresql...
View Articlehow to identify rogue consumer
I have seen very high "Fetch-Consumer-RequestsPerSec" (like 15K) per broker in a relatively idle cluster. My hypothesis some misbehaving consumer has a tight polling loop without any back-off logic...
View ArticleIs kafka suitable for our architecture?
Hi I just came across Kafta when I was trying to find solutions to scale our current architecture. We are currently downloading and processing 6M documents per day from online and social media. We have...
View Articlecreate topic in multiple node kafka cluster
Hi, All I setup a 3-node kafka cluster on top of 3-node zk ensemble. Now I launch 1 broker on each node, the brokers will be randomly distributed to zk ensemble, see DO-mq-dev.1 [zk:...
View Articleincluding KAFKA-1555 in 0.8.2?
Hi, Everyone, I just committed KAFKA-1555 (min.isr support) to trunk. I felt that it's probably useful to include it in the 0.8.2 release. Any objections? Thanks, Jun
View ArticleClarification about Custom Encoder/Decoder for serialization
Hi All, I wanted to get some clarification on Kafka's Encoder/Decoder usage. Lets say I want to implement a custom Encoder. public class CustomMessageSerializer implements Encoder<MyCustomObject>...
View ArticleAuto Purging Consumer Group Configuration
Hi Kafka, We have lots of lingering console consumer group people have created for testing or debugging purpose for one time use via bin/kafka-console-consumer.sh. Is there auto purging that clean...
View ArticleScala+Kafka puzzler
Hello Apache Kafka community, Attached trivial Maven built project with Kafka code fails to compile, with error: "class Broker in package cluster cannot be accessed in package kafka.cluster" Does...
View Articlemount of messages produced and those consumed are not equal
Dear folks, I have run some basic tests using kafka. I have started 7 brokers, created a topic with 21 partitions and 3 replicas. 2 producers are started with 5 threads each and pushed 2419200 messages...
View ArticleStarting KafkaMetricsReporter from Kafka Consumer/Producer
In ProducerPerformance class we use CSVMetricsReporter for metrics reporting. Which I think is actually started with the help of below function: KafkaMetricsReporter.startReporters(verifiableProps)...
View ArticleInstallation and Java Examples
I am new to Kafka and very little familiarity with Scala. I see that the build requires "sbt" tool, but do I also need to install Scala separately? Is there a detailed documentation on software...
View Articlekafka-web-console
Hi, all I am installing kafka-web-console on ubuntu server, when I sbt package it, it stuck on waiting for ivy.lock root [ at ] DO-mq-dev:/home/stuser/kafkaprj/kafka-web-console# sbt package Loading...
View Articlekafka java api (written in 100% clojure)
Hi, Just thought I'll put this out for the kafka community to see (if anyone finds it useful great!!). Kafka-fast is 100% pure clojure implementation for kafka, but not just meant for clojure because...
View Article