Using the kafka dissector in wireshark/tshark 1.12
I'd seen references to there being a Kafka protocol dissector built into wireshark/tshark 1.12, but what I could find on that was a bit light on the specifics as to how to get it to do anything -- at...
View ArticleBlocking Recursive parsing from kafka.consumer.TopicCount$.constructTopicCount
Hi All, We have a typical cluster of 3 kafka instances backed by 3 zookeeper instances (kafka version 0.8.1.1, scala version 2.10.3, java version 1.7.0_65). On consumer end, when some of our consumers...
View ArticleCorrect way to handle ConsumerTimeoutException
Folks, I am using consumer.timeout.ms to force a consumer jump out hasNext call, which will throw ConsumerTimeoutException. It seems that upon receiving this exception, the consumer is no longer usable...
View ArticleConsuming messages from Kafka and pushing on to a JMS queue
Hi, We have an application that currently uses Camel to forward a JMS message from HornetQ to multiple consumers (JMS queues). The one challenge we have is handling failure of one of the consumers....
View Articleconsumer read from specific partition
Hi, Suppose I have N partitions. I would like to have X different consumer threads ( X < N) read from a specified set of partitions. How can I achieve this? Thanks, Josh
View ArticleHow many threads should I use per topic
Hey, Guys, I am using the high level consumer. I have a daemon process that checks the lag for a topic. Suppose I have a topic with 5 partitions, and partition 0, 1 has lag of 0, while the other 3 all...
View ArticleHow to perform a controlled shutdown for rolling bounce?
Running 0.8.1 and am unable to do a controlled shutdown as part of a rolling bounce. Is this the primary reference for this task?...
View ArticleWriting to Kafka
Hi, I started using kafka with Samza. I’m trying to run a test that is supposed to create messages and write to a kafka topic. In my test I start writing a small amount of messages and it should grow...
View ArticleConsumer sensitive expiration of topic
Hi, I'm playing around with Kafka with the idea to implement a general purpose message exchanger for a distributed application with high throughput requirements (multiple hundred thousand messages per...
View ArticleJMSXGroupID clustered message grouping
Hi, When using multiple partitions and consumers, can Kafka ensure that JMS messages with the same JMSXGroupID are handled by the same consumer? Or does this have to be implemented when writing the...
View ArticleHow try out the new producer?
Hi, I want to try out the new producer api (org.apache.kafka.clients.producer.KafkaProducer) but found that it's not in the published jar. What's the best way to get it? Build from source from the...
View ArticleEmbedded Kafka/Zookeeper for unit testing
Hi all, If someone is searching for some up-to-date examples of embedded kafka/zookeeper servers, like for unit testing, maybe will find this short blog post useful:...
View ArticleZkClient bug can bring down broker/consumer on zookeeper push in EC2 environment
Recently, we found the serious ZkClient bug, actual Apache Zookeeper client bug, which can bring down broker/consumer on zookeeper push. We're running kafka and zookeeeper in AWS EC2 environment....
View ArticleKeep on getting kafka.common.OffsetOutOfRangeException: Random times
Hi Team, Of late I am facing strange issue w.r.t Kafka. Random times I keep on getting these strange errors while consuming the topic: kafka.common.OffsetOutOfRangeException: Request for offset 19 but...
View Articlekafka contrib/hadoop-consumer,producer
Hi, I would like to know the status regarding contrib/hadoop-consumer,hadoop-producer code. Is this considered to be deprecated or there is still interest in these projects. Is casmus the current...
View ArticleHow to shutdown/restart kafka cluster properly
Hi, Sorry if this had been answered before. Although, I couldn't find any information besides "controlled shutdown of broker", which, I believe not fully applies here. Could anyone suggest what would...
View ArticleUsing kafka in non million users environment
Hello, I'm managing a study to explore possibilities for migrating a monolith architecture IT to a service oriented one. That said, the company i'm working for is not a web based company, so the number...
View Articlekafka high level consumer - threads guaranteed to read a single partition?
Hi, For the kafka high level consumer, if I create exactly the number of threads as the number of partitions, is there a guarantee that each thread will be the only thread that reads from a particular...
View Articlehow is the partitioner working when compression is enabled?
I am using Kafka 0.8 and message compression can only be enabled on producer side. However one batch of messages (decided by the producer configuration batch.num.messages) is compressed as a single...
View Article