shallow iteration
Hi How to enable shallow iterations for consumers? I checked http://kafka.apache.org/08/configuration.html but nothing has been found about it. Thanks. Regards, Libo
View ArticleProducer only finding partition on 1 of 2 Brokers, even though ZK shows 1...
(version 0.7.2) For some reason, my producers are only picking up the partition on 1 of my 2 brokers. I've been digging through the code, and I don't see any issues given the state of my ZK nodes. The...
View ArticleCompression in kafka 0.8
Good evening. I would like to preserve logs in kafka for 2 days. Due to the size of data it must be compressed otherwise it will take enormous amount of disk space. Its my understanding that the only...
View Article2 kafkas 1 zookeeper
We are interested in having two kafka clusters (one 0.7.2, one 0.8) share the same 3 node zookeeper tier (3.4.3). Has anyone else tried something like this in the past? Should we anticipate any issues?...
View Articleconsumer offset (0.7)
Hey guys, Is there any way to tell where the consumer offset is in terms of the total size of the topic or topic-partition? I know i can read the offset itself out of Zookeeper, but that doesn't seem...
View ArticleMonitor Kafka Stats
Hi I am trying to get some parameters from kafka like Data produced per second, Data Consumed per second etc. There is a java api in Kafka "kafka.network.SocketServerStats" that i am using for my...
View Articlebug report
The publisher tried to publish 10G data (msg size 10k) to 3 brokers. This error occurred so frequently and caused big delay. [2013-06-06 11:13:29,491] INFO Shutting down producer...
View Articlepackage error
I am quite unfamiliar with compiling with sbt package. What could be my issue here? It seems like the scala library is wrong, but where do I look for the scala jar? thanks, rob "$ ./sbt package [info]...
View ArticleArguments for Kafka over RabbitMQ ?
Hi -- I am preparing to make a case for using Kafka instead of Rabbit MQ as a broker-based messaging provider. The context is similar to that of the Kafka papers and user stories: the producers publish...
View ArticlePostie - Kafka Consumer Framework
Hi all! This excellent presentation on StumbleUpon's implementation of Kafka and other Open Source technologies mentions 'Postie' as a 'Kafka Consumer Framework':...
View Articlejava.lang.NumberFormatException: null - when producer tries to send message...
Hi, I am getting exception when Producer tries to send messages to Kafka queue. It tries to connect to zookeeper instances and then throws exception. Here's initial log - 06 Jun 2013 15:17:29,541...
View ArticleDilemma - ZK consumer problems - upgrade to 0.8?
Hey guys, I sincerely apologize if this has been covered before, I haven't quite found a similar situation. We are using Kafka 0.7.2 in production, and we are using the ZK high level Scala consumer....
View ArticleDilemma - ZK consumer woes - upgrade to 0.8?
[ Sorry if this mail is duplicated, this is my fourth try sending this message] Hey guys, I sincerely apologize if this has been covered before, I haven't quite found a similar situation. We are using...
View ArticleKafka Messages
Im a bit confused on the concept of a "message" in Kafka. How does this differ, if at all, from a message in RabbitMQ? It seems to me that Kafka is better suited for very write intensive "messages"...
View Articlerebalance issues from mirrormaker
Hi, I come across several critical issues when benchmarking mirrormakers. 1 If a topic has N partitions on source cluster, after mirroring, on the destination cluster it has only one partition. 2 To...
View Articleout of disk space
Hi, The volume I used for kafka has about 100G space. I set log.retention.bytes to about 60G. But at some point, the disk was full and the processes crashed. I remember other people reported the same...
View Articlelog file corruption when reading older messages
We've been running Kafka 0.7.0 in production for several months and have been quite happy. Our use case to date has been to pull from the head of our topics, so we're normally consuming within seconds...
View ArticleKakfa Failures and Recovery
Hi, We are using 0.8 version of Kafka and are planning for high availability testing with replication. While the entire scheme to enable the cluster to be highly available is clear, I wanted to get...
View ArticleEvents stored in Apache Web Logs - how best to get these into Kafka?
I have a lot of user event data being sent to an apache web server and written to web logs. Unfortunately I don't control this flow, but do I have access to the server the logs are being written to. I...
View ArticleKafka 0.8 Maven and IntelliJ
I downloaded the latest 0.8 snapshot and I want to build using Maven: ./sbt make-pom Generates a bunch of pom.xml files but when I try to open one of them in IntelliJ they are not recognized. Do I need...
View Article