Dynamically adding Kafka brokers
Hi, How do people handle situations, and specifically the broker.id property, where the Kafka (broker) cluster is not fully defined right away? Here's the use case we have at Sematext: * Our software...
View ArticleIssue with async producer
Hi, We are using an async producer to send data to kafka. The load the sender handles is around 250 rps ,the size of a message is around 25K. The configs used in the producer are :...
View ArticleAdvertised host name
Hi, While setting up a cluster, I realised that Kafka uses host names to communicate between brokers. In my case, I don't have a DNS server to keep track of host names, and set up the host names...
View ArticleTuning replication
Good day all, We're running a good sized Kafka cluster, running 0.8.1, and during our peak traffic times replication falls behind. I've been doing some reading about parameters for tuning replication,...
View Articlequeued.max.message.chunks impact and consumer tuning
Hi Kafka Dev Team, It seems that Maximum buffer size is set to 2 default. What is impact of changing this to 2000 or so ? This will improve the consumer thread performance ? More event will be buffered...
View ArticleKafka 0.8.1.1 replication issues
Hi Kafka users! I was just migrating a cluster of 3 brokers from one set of EC2 instances to another, but ran into replication problems. The method of migration used is that of stopping one broker and...
View ArticleProducer timeout setting not respected
Hi all, I've been investigating how Kafka 0.8.1.1 responds to the scenario where one broker loses connectivity (due to something like a hardware issue or network partition.) It looks like the brokers...
View ArticleOffsetOutOfRange Error
Hello, We are using kafka version 0.8.1 and the python kafka client. Everything has been working fine and suddenly this morning I saw a OffsetOutOfRange on one of the partitions. (We have 20 partitions...
View Articleconsumer ack for high-level consumer?
Hi, I am a new to Kafka. In my understanding, high-level consumer ( ZookeeperConsumerConnector) changes offset when message is drawn by ConsumerIterator. But I would like to change offset when message...
View ArticleLocation of Logging Files/How To Turn On Logging For Kafka Components
Background: I have searched for a while online, and through the files located in the kafka/logs directory, trying to find where kafka writes log output to in order to debug the SimpleProducer I wrote....
View ArticleHow costly is Re balancing of partitions for a topic
Hello, I am trying to come up with a design for consuming from Kafka. *I am using 0.8.1.1 version of Kafka. *I am thinking of designing a system where the consumer will be created every few seconds,...
View ArticleProducer and Consumer properties
Hi Dudes, I would like to know if the producer and consumer’s properties files into the config folder should be configured. I have configured only the server.properties, is it enough? I am doing some...
View ArticleStoring data in kafka keys
Hi, It looks like it is a general practice to avoid storing data in kafka keys. Some examples of this: Camus, Secor both not using keys. Even such a swiss-army tool as kafkacat doesn't seem to have the...
View ArticleConsumer lag keep increasing
Hey Guys, I have a really simply storm topology with a kafka spout, reading from kafka through high level consumer. Since the topic has 30 partitions, we have 30 threads in the spout reading from it....
View ArticleNo longer supporting Java 6, if? when?
This has been coming up in a lot of projects and for other reasons too I wanted to kick off the discussion about if/when we end support for Java 6. Besides any API we may want to use in >= 7 we also...
View ArticleHigh CPU usage of Crc32 on Kafka broker
Hi, Using flight recorder, we have observed high CPU usage of CRC32 (kafka.utils.Crc32.update()) on Kafka broker. It uses as much as 25% of CPU on an instance. Tracking down stack trace, this method is...
View Article"metric.reporters" is not working
Hi When I set up props.put("metric.reporters", Lists.newArrayList(ServoReporter.class.getName())); I got the following error: org.apache.kafka.common.config.ConfigException: Unknown configuration...
View ArticleKafka Release timelines
Hi, I found the future release plan wiki here <https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan> . I see that 0.8.2 is still in beta even though it was stated for September....
View ArticleInformation
Hi Guys Anyone could explain me this information? 208K), 0.0086120 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] 2014-11-06T12:20:55.673+0100: 1256.382: [GC2014-11-06T12:20:55.674+0100: 1256.382:...
View Articlecorrupt recovery checkpoint file issue....
Hi, We recently had a kafka node go down suddenly. When it came back up, it apparently had a corrupt recovery file, and refused to startup: 2014-11-06 08:17:19,299 WARN [main] server.KafkaServer -...
View Article