Kafka lost data
Hello folks, I recently noticed our message amount in kafka seems to have dropped significantly. I didn't see any exception on my consumer side. Since producer is not within my control, I am trying to...
View Articlequestion about async publishing for 0.8.1
Hi team, I am testing async publishing + acknowledgement. Assume all settings are the default and queue.buffering.max.messages is 10k. I use a simple for loop to publish 100k messages. In the process,...
View ArticleApache Kafka 0.8.2-beta Released
The Apache Kafka community is pleased to announce the beta release for Apache Kafka 0.8.2. The 0.8.2-beta release introduces many new features, improvements and fixes including: - A new Java producer...
View ArticleWhere Compression/Decompression happens
Hi Kafka Team, Is Compression happening on Producer Side (on application thread meaning thread that call send method or background Kafka thread ) and where does decompression Consumer side ? Is there...
View ArticleHigh Level Consumer and Close with Auto Commit On
Hi Kafka Team, What is expected behavior when you close *ConsumerConnector* and auto commit is on ? Basically, when auto commit interval is set to 5 seconds and shutdown is called (before 5 seconds...
View ArticleApache Kafka Consumers in java 8
Hi All, Can you please share your experiences of running Kafka Consumers/producers with Java 8 ?. Thanks, Balaji
View ArticleConsumer cannot find leader after leadership changes on broker side
After executing PreferredReplicaLeaderElectionCommand on broker instance, we observed one of the consumers cannot find the leadership and stopped consuming. The following exception is all over the log...
View ArticleConsuming a large number of partitions in a deterministic way
I am planning to use the current Java API and have the following use case: i) A single topic with about 1024 partitions. ii) A number of processes that want to consume these partitions in a...
View ArticleKafka producer error
I keep seeing these errors in my code that is just trying to send some data using an AsyncProducer: kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at...
View ArticleSimpleConsumer fetching the last read offset
Does the SimpleConsumer provide any facility to record what offset in a partition was last processed by a consumer? From the example I have seen the use of kafka.api.OffsetRequest.EarliestTime() and...
View ArticleCan both uncompressed data and compressed data reside in the same topic...
Hey Guys, If at the beginning, I configure the producer to not use compression and produce a number of message to a topic. Then later, I reconfigure the producer to use compression and produce another...
View ArticleKafka Cluster disaster decovery
Hi All, I’m new to Kafka, please direct me to the right path if it is a duplicate question. Basically I deployed Kafka to a 4 machine cluster, what if the whole cluster went down, does kafka provide...
View ArticleError using migrationtool for upgrading 0.7 to 0.8
Hi I'm trying to upgrade a 0.7 kafka cluster, but I'm getting an error: I created the file migrationToolConsumer.properties with just theinfo to connect to the old cluster: _____...
View ArticleConsumer keeps looking connection
Hello Folks, I am using Highlevel consumer, and it seems to drop connections intermittently: 2014-11-01 13:34:40 SimpleConsumer [INFO] Reconnect due to socket error: Received -1 when reading from...
View Articlekafka.message.InvalidMessageException: Message is corrupt
Hi We are experimenting with running kafka server on a windows machine, but keep getting exeptions when producing a lot of messages (in the neighborhood of 1 million).....
View Articlequestion about console producer behavior
Hi It appears to me that, when I have opened a console producer, it will not notice the topic’s partition change? say I have a topic with 3 partitions. And I opened a console producer to produce data...
View ArticleSpark Kafka Performance
Hi Guys, Anyone could explain me how to work Kafka with Spark, I am using the JavaKafkaWordCount.java like a test and the line command is: ./run-example...
View Articlecorrupt message
When running a C# producer against a kafka 0.8.1.1 server running on a virtual linux (virtualbox, Ubuntu) I keep getting the following error: [2014-11-03 15:19:08,595] ERROR [KafkaApi-0] Error...
View Articlekafka producer example
Hi, All I am running the kafka producer code: import java.util.*; import kafka.javaapi.producer.Producer; import kafka.producer.KeyedMessage; import kafka.producer.ProducerConfig; public class...
View Article