Message details
Hi, We are experimenting Kafka for a MQ use-case. We found it very useful but couldn't find the following info from the documentation: I have a consumer logic which can say that a message consumption...
View ArticleKafka-Storm Run-time Exception
I am using kafka with storm. I am using maven to build my topology and I am using scala 2.9.2 same as I am using kafka_2.9.2_0.8.1. Topology build perfectly using maven. But hwn I submit the topology...
View ArticleTrouble with snappy and SimpleConsumer
Hi, I am using kafka_2.9.2-0.8.1 and snappy-java-1.1.0.1.jar I have been able to successfully use gzip with the same library. however “snappy” doesn’t work in consumer side. Producer is able to send...
View ArticleGetting the KafkaStream ID
Hello folks, I’m using Kafka 0.8.0 with the high level consumer and I have a situation where I need to obtain the ID for each of the KafkaStreams that I create. The KafkaStream class has a method...
View Articlemessage.max.bytes and mirrormaker tool
I've tried to run mirrormaker tools in async mode and I get WARN Produce request with correlation id 263 failed due to [benchmark1,30]: kafka.common.MessageSizeTooLargeException...
View ArticleNoReplicaOnlineException with 0.8.1.1
Hi, I am running Kafka 0.8.1.1 with one Zookeeper and one broker. I created a partition 'test2' as below: /kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3...
View ArticleAbout peculiar scenario in kafka camus consumer
Hi I have observed a peculiar scenario in production environment in which a mapper task for a particular topic-partition combination always fails with the exception 'Task attempt failed to report...
View ArticleSync Producer
Hi, How to use a sync producer with a KeyedMessage<String, String>. The example in the documentation points to the Async Producer. What exceptions will be thrown if the producer.send() fails?...
View ArticleHandling un-decodable messages (Kafka 0.8.0)
I have a custom Decoder for my messages (Thrift). I want to be able to handle "bad" messages that I can't decode. When the ConsumerIterator encounters a bad message, the exception thrown by my Decoder...
View ArticleStrange partitioning behavior with 0.8.1.1
Hi, This is with 0.8.1.1 and I ran the command line console consumer. I have one broker, one producer and several consumers. I have one topic, many partitions m, many consumers n, m=n , one consumer...
View ArticleError: Could not find or load main class kafka.perf.ProducerPerformance
Hi, Is the perf module bundled with 0.8.1.1 ? When i try to run "kafka-producer-perf-test.sh" I see *Error: Could not find or load main class kafka.perf.ProducerPerformance* I then tried to do a...
View Articleorg.apache.zookeeper.KeeperException$BadVersionException
Hi, team. I’m using 0.8.1. I found some strange log repeatedly on server.log in one of my brokers and it keeps logging until now. server.log...
View ArticleLeaderNotAvailableException in 0.8.1.1
Hi, I am running a cluster with a single broker, the performance producer script and 3 consumers. On a fresh start of the cluster , the producer throws this exception. I was able to run this cluster...
View Articlespam of INFO kafka.network.Processor - Closing socket connection to...
Hi all! I have 3 brokers and I keep getting this message: [kafka-processor-9092-0] INFO kafka.network.Processor - Closing socket connection to /172.23.44.4. or [kafka-processor-9092-1] INFO...
View Articlekafka splunk integration
Hello All, Is there a kafka splunk loader/consumer/subscriber? I read in the powered-by page for kafka, that square outputs kafka logs to splunk. Does anyone know how this is done? thanks, Clay
View Articleec2 suggestion setup for a minimum kafka setup; zookeeper is bad in ec2?
For those of you hosting on ec2, could someone suggest a "minimum" recommended setup for kafka? i.e. the # and type of instance size that you would say is the bare minimum to get started with kafka in...
View ArticleCould you pls add my email to the mailing list
Hi, Good Morning !! Could you please add my email to the Kafka mailing list My Email id solocts [ at ] gmail.com
View Articlescala based service/deamon consumer example
Is there a simple example (scala preferred) where there is a consumer that is written to run as a deamon i.e. it keeps as open connection into a broker and reads off new messages
View ArticleNew Clojure library for Kafka
Hi all, I was working with Kafka the other day and I wanted to use my favorite language but the "official" driver was rather cryptic and there too much black magic going on, so I decided to rewrite it...
View ArticleKafka High Level Consumer Fail Over
Hi Kafka Dev Team/ Users, We have high level consumer group consuming from 32 partitions for a topic. We have been running 48 consumers in this group across multiple servers. We have kept 16 as back-up...
View Article