no zk connection timeout in consumer config of kafka 0.7?
Hi, In http://kafka.apache.org/08/configuration.html, there are 2 parameter about zk in consumer. zookeeper.session.timeout.ms 6000 Zookeeper session timeout. If the consumer fails to heartbeat to...
View ArticleKafkaSpout
Hi all, I am using a kafka spout for reading data from a producer into my storm topology. On the kafkaconfig : 1. When i set Kafka.forcefromstart = true .. it read from the beginning , however if i...
View ArticleNon-blocking High-Level Consumer
Hi, Is it possible to read all available messages with HLC in a non-blocking way? E.g. read all messages and not wait for more messages to appear in the topic. As far as I understand, currently one has...
View ArticleMBeans, dashes, underscores, and KAFKA-1481
Hi, *Problem:* Some Kafka 0.8.x MBeans have names composed of things like <consumer group>-<topic>-<metric name>. Note how dashes are used as delimiters. When <consumer group>...
View ArticleWhy is Scala 2.9.2 version of kafka recommended?
On Kafka download page we saw - "We build for multiple versions of Scala. This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version...
View Articlezookeeper upgrade or remove zookeeper dependency
Hi, I could see kafka is using zookeeper 3.3.4. For my integration purpose, I want to use curator, which requires a higher version than 3.3.4 even in its lowest version. I there any plan to bump up...
View ArticleDifferent partitioning between new producer and old producer
The major motivation of adopting new producer before it's released, old producer is showing terrible throughput of cross-regional kafka mirroring in EC2. Let me share numbers. Using iperf, network...
View ArticleDisactivating Yammer Metrics Monitoring
Hi all! We are using yammer metrics to monitor some parts of our system. Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data getting in our graphite server and from what I saw, it...
View ArticleCPU Usage Spike to 100% when network connection is lost
Hi Kafka Dev team, I see my CPU spike to 100% when network connection is lost for while. It seems network IO thread are very busy logging following error message. Is this expected behavior ? 2014-09-17...
View ArticleInteraction of retention settings for broker and topic plus partitions
Hello all! I'm curious about the interaction of server and topic level retention settings. It's not clear to me the precedence of the follow: - broker's default log.retention.bytes - topic's...
View ArticleHandling errors in the new (0.8.2) Java Client's Producer
I am trying to understand the best practices for working with the new (0.8.2) Producer interface. We have a process in a large server that writes a lot of data to Kafka. However, this data is not...
View Articlesubscribe kafka mail list
Hi, I am interested in Kafka, I want to subscribe kafka mail lists, Thanks.
View ArticleCopying messages from a single partition topic to a multi-partition topic
Hi, We have an interesting problem to solve due to a very large traffic volumes on particular topics. In our initial system configuration we had only one partition per topic, and in in a couple of...
View ArticleHow to use RPC mechanism in Kafka?
Hi, Please tell me how to use request/response mechanism in kafka? Thanks Lavish Goel
View ArticleQuestions re: partition reassignment
Good day, We're currently running a Kafka cluster in our staging environment, testing everything out and so far it runs fairly smoothly. One thing that we're currently looking into is scaling a cluster...
View ArticleBusy CPU while negotiating contentBuffer size at...
Folks, Recently in one of our SimpleConsumer based client applications (0.8.1.1),we spotted a very busy CPU with almost no traffic in/out from the clientand Kafka broker (1broker+1zookeeper) (the stack...
View ArticleLog 4j Conversion Pattern not working kafka.producer.KafkaLog4jAppender in...
Hello , I am trying to use KafkaLog4jAppender to write to kafka and also a file appender to write to a file. The conversion pattern works for the file, but for the kafka appender it is not working The...
View ArticleRepost: Busy CPU while negotiating contentBuffer size at...
Note: Re-posting the older message from another account due to formatting issues. Folks, Recently in one of our SimpleConsumer based client applications (0.8.1.1), we spotted a very busy CPU with...
View ArticleOffset manager movement (due to change in KAFKA-1469)
I just wanted to send this out as an FYI but it does not affect any released versions. This only affects those who release off trunk and use Kafka-based consumer offset management. KAFKA-1469 fixes an...
View ArticleProducer errors (failed to send producer request, failed to send requests for...
I have a test data set of 1500 messages (~2.5 MB each) that I'm using to test Kafka throughput. I'm pushing this data using 40 Kafka producers, and I'm losing about 10% of the message on each trial....
View Article