kafka test jars in sbt?
Hello, I want to use the kafka_2.10-0.8.2-beta-test.jar in my Scala project. It can be found here: http://repo.maven.apache.org/maven2/org/apache/kafka/kafka_2.10/0.8.1.1/ In my build.sbt I write the...
View ArticleConsumer and Producer configs
Hi Guys, How could I use the Consumer and Producer configs in my Kafka environment? Thanks
View ArticleOffsetOutOfRange Error
I dug deeper and saw this during normal operation: In the kafka broker log: [2014-11-03 21:39:25,658] ERROR [KafkaApi-8] Error when processing fetch request for partition [activity.stream,5] offset...
View ArticleSpark and Kafka
Hi Guys, I am doing some tests with Spark Streaming and Kafka, but I have seen something strange, I have modified the JavaKafkaWordCount to use ReducebyKeyandWindow and to print in the screen the...
View ArticleAnnouncing Confluent
Hey all, I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a company around Kafka called Confluent. We are planning on productizing the kind of Kafka-based real-time data platform we...
View ArticleRe:Announcing Confluent
Congratulations Jay, Jun and Neha. Great news. Wish you good luck!!! From: users [ at ] kafka.apache.org To: dev [ at ] kafka.apache.org, users [ at ] kafka.apache.org At: Nov 6 2014 13:28:51 Hey all,...
View ArticleStrategies for high-concurrency consumers
Hi all, We are building a system that will carry a high volume of traffic (on the order of 2 billion messages in each batch), which we need to process at a rate of 50,000 messages per second. We need...
View Articlenulls found in topic, created by recovery?
Hi all. I saw something weird yesterday on our "leaf" instances which run kafka 0.7.2 (and mirror to kafka 0.8 via our custom code). I fully realize everyone's instinctual response is "upgrade,...
View ArticleOffsetOutOfRange errors
Hello, I understand what this error means, just not sure why I keep running into it after 24-48 hrs of running fine consuming > 300 messages / second. What happens when a kafka log rolls over and...
View ArticleAdding replicas to existing topic cause data loss in some partitions
I have a kafka cluster, every topic in it has only one replica. Recently I extend every topic with 2 replicas. Most topics work fine, but some large topics have some problems with part of partitions....
View ArticleInterrupting controlled shutdown breaks Kafka cluster
Hi all, My team has observed that if a broker process is killed in the middle of the controlled shutdown procedure, the remaining brokers start spewing errors and do not properly rebalance leadership....
View ArticleConsumer thread dies
Hi, I have a scenario where I have 1 partition and 1 consumer group having 2 consumer threads running say C1 and C2. Since there is only one partition for a given topic, say C1 is holding that...
View ArticleAdd partitions with replica assignment in same command
I am trying to figure out how to add partitions and assign replicas using one admin command. I tried kafka.admin.TopicCommand to increase the partition number from 9 to 12 with the following options:...
View Articlepowered by kafka
Exponential @exponentialinc is using kafka in production to power the events ingestion pipeline for real time analytics and log feed consumption. Please post on powered by kafka wiki -...
View ArticleIssues Running Kafka Producer Java example
Hello Champs, I am trying to run first java producer example. Upon running this example, producer successfully sends the message, at least it looks like it does, there is no java dump But trying to...
View Articleexpanding cluster and reassigning parititions without restarting producer
Hi All, I recently had an issue producing from python where expanding a cluster from 3 to 5 nodes and reassigning partitions forced me to restart the producer b/c of KeyError thrown. Is this situation...
View ArticleOne question about "New Producer Configs"
Dear sir or madam, There is one question when I'm using Kafka: From the "documentation:New Producer Configs" ,I get that acks=n means the number of acknowledgments the producer requires the leader to...
View ArticleError in fetch Name. How to recover broken node?
Hi, i've got a 2-machine kafka cluster. For some reasons after a restart the second node won't start. i get tons of "Error in fetch Name" until I get a final "Too many open files". How do i start...
View ArticleJavaKafkaWordCount not working under Spark Streaming
I am embarrassed to admit but I can't get a basic 'word count' to work under Kafka/Spark streaming. My code looks like this. I don't see any word counts in console output. Also, don't see any output in...
View Articlechange retention for a topic on the fly does not work
Hey guys, i am using kafka_2.9.2-0.8.1.1 bin/kafka-topics.sh --zookeeper localhost:2182 --alter --topic my_topic --config log.retention.hours.per.topic=48 It says: Error while executing topic command...
View Article