Fetch Response for invalid offset returns success.
Good afternoon. I am writing my own client that is based on simple consumer. I implemented method that is based on example in the kafka documents that sends request to kafka to find an offset. I am...
View ArticleKafka Sync vs. Async etc.
Hi: If someone's intention is to implement a producer such that: 1. The call to send() blocks until the write is successful on the configured number or replicas or max num of retries. 2. The call to...
View Articlepython brond - brod.base.OffsetOutOfRange: Code: 1
How do I fix? In python.... for offset, message in brod.Kafka.fetch(kafka,pixel_topic, offset=0): print message File "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py", line...
View ArticleKafka Web Console
Announcing the first release of Kafka Web Console: https://github.com/claudemamo/kafka-web-console/tree/v1.0. Tested on Chrome. Enjoy, Claude
View Articlefidelity of offsets when mirroring
Hi, I have a question about mirroring. I would like to create a highly available Kafka service that runs on AWS and can survive an AZ failure. Based on what I've read, I plan to create a Kafka cluster...
View ArticleAccessing kafka from different geo location
Hi, We have a use case where we have the kafka cluster in a different datacenter from the consumers. The kafka cluster is in the same data center where producers are. I have heard that zookeeper could...
View ArticleFailed in automatically creating topic
Hi, According to the auto.create.topics.enable description, 1). attempts to produce, 2), consume, 3), fetch metadata for a non-existent topic will automatically create the topic with default config. I...
View ArticleFailed in automatically creating topic
Hi, According to the auto.create.topics.enable description, 1). attempts to produce, 2), consume, 3), fetch metadata for a non-existent topic will automatically create the topic with default config. I...
View ArticleUsability
Hey guys, It would be good to tag any JIRA for something which is an confusing or annoying with the "usability" tag. I am trying to get a list of all these together so we can take a wack at some of...
View ArticleReg Partition
Hi I have a doubt on the parallelism. Why the number of parallel consumer consuming messages from a topic is restricted on the number of partitions configured for a topic? Why should this be the case....
View ArticleKafka Producer load distribution
Hi, I am new with kafka and using kafka 0.8 to build a distributed queuing system in amazon web service cluster. I have 4 machines Z1, B1, B2 and B3. 1 Zookeeper instance is running on Z1 and 3...
View ArticleDocumentation for the upcoming 0.8.1 release
Hey guys, I took a stab at updating the docs for the 0.8.1 release. In particular, I added a section on log compaction: http://kafka.apache.org/081/documentation.html#compaction I also updated the...
View ArticleQuestion concerning partitionNumber and Key
Hi, I have 2 questions about the partition number and key. 1. The produceRequest will explicitly include a partitionNumber, and messageSet which contains messages with key(can be NULL, used to...
View ArticleConsumer Multi-Fetch
I understand replication uses a multi-fetch concept to maintain the replicas of each partition. I have a use case where it might be beneficial to grab a “batch” of messages from a kafka topic and...
View Articleconsumer reaction to auto-create topics
Hi there, If i start a consumer in a non-existent topic (auto-create true) before the producer, the consumer never gets the messages. These are the steps: 1) kafka-console-consumer --topic newTopic (it...
View Articlehow to find start offset of messages stored at kafka broker
Hi, Is there a way to get the start offset of messages present at current point in kafka broker, given the topic and partition. I am using high-level consumer, and I want to replay messages from a...
View ArticleAre offsets unique, immutable identifiers for a message in a topic?
Hi, I have the following problem: My Kafka consumer is consuming messages, but the processing of the message might fail. I do not want to retry until success, but instead want to quickly consume the...
View ArticleAnouncing Kafka Offset Monitor 0.1
Hello everyone, at Quantifind, we are big users of Kafka and we like it a lot! In a few use cases, we had to figure out if a queue was growing and how its consumers were behaving. There are a few...
View ArticleApache Kafka Use Case at WalmartLabs
We are planning to use Apache Kafka to replace Apache Fume for mostly as log transport layer. Please see the attached image which is similar use case ( and deployment architecture ) at Linkedin...
View ArticleQuestion on StateChangeFailedException
How can I prevent these errors coming? When I created 9 partitions on 3 instances and 2 replication-factor, I didn't have any error. But when I created 36 partitions on 12 instances, I got these...
View Article