Message with Custom Partition Logic
HI Kafka Dev Team, We have to aggregate events (count) per DC and across DCs for one of topic. We have standard Linked-in data pipe line producers --> Local Brokers --> MM --> Center Brokers....
View ArticleIssue with 240 topics per day
Folks, Is there any potential issue with creating 240 topics every day? Although the retention of each topic is set to be 2 days, I am a little concerned that since right now there is no delete topic...
View ArticleConsumer Parallelism
Hi, We are using the following method on ConsumerConnector to get multiple streams per topic, and we have multiple partitions per topic. It looks like only one of the runnable is active through a...
View Articlekafka TestUtils createBrokerConfig issue
Trying to write a unit test case for Kafka, and stuck with strange createBrokerConfig issue. on TestUtils.createBrokerConfigs(1) it gives me compilation error The method createBrokerConfigs(int,...
View ArticleUsing the kafka dissector in wireshark/tshark 1.12
I'd seen references to there being a Kafka protocol dissector built into wireshark/tshark 1.12, but what I could find on that was a bit light on the specifics as to how to get it to do anything -- at...
View ArticleBlocking Recursive parsing from kafka.consumer.TopicCount$.constructTopicCount
Hi All, We have a typical cluster of 3 kafka instances backed by 3 zookeeper instances (kafka version 0.8.1.1, scala version 2.10.3, java version 1.7.0_65). On consumer end, when some of our consumers...
View ArticleCorrect way to handle ConsumerTimeoutException
Folks, I am using consumer.timeout.ms to force a consumer jump out hasNext call, which will throw ConsumerTimeoutException. It seems that upon receiving this exception, the consumer is no longer usable...
View ArticleConsuming messages from Kafka and pushing on to a JMS queue
Hi, We have an application that currently uses Camel to forward a JMS message from HornetQ to multiple consumers (JMS queues). The one challenge we have is handling failure of one of the consumers....
View Articleconsumer read from specific partition
Hi, Suppose I have N partitions. I would like to have X different consumer threads ( X < N) read from a specified set of partitions. How can I achieve this? Thanks, Josh
View ArticleHow many threads should I use per topic
Hey, Guys, I am using the high level consumer. I have a daemon process that checks the lag for a topic. Suppose I have a topic with 5 partitions, and partition 0, 1 has lag of 0, while the other 3 all...
View ArticleHow to perform a controlled shutdown for rolling bounce?
Running 0.8.1 and am unable to do a controlled shutdown as part of a rolling bounce. Is this the primary reference for this task?...
View ArticleWriting to Kafka
Hi, I started using kafka with Samza. I’m trying to run a test that is supposed to create messages and write to a kafka topic. In my test I start writing a small amount of messages and it should grow...
View ArticleConsumer sensitive expiration of topic
Hi, I'm playing around with Kafka with the idea to implement a general purpose message exchanger for a distributed application with high throughput requirements (multiple hundred thousand messages per...
View ArticleJMSXGroupID clustered message grouping
Hi, When using multiple partitions and consumers, can Kafka ensure that JMS messages with the same JMSXGroupID are handled by the same consumer? Or does this have to be implemented when writing the...
View ArticleHow try out the new producer?
Hi, I want to try out the new producer api (org.apache.kafka.clients.producer.KafkaProducer) but found that it's not in the published jar. What's the best way to get it? Build from source from the...
View ArticleEmbedded Kafka/Zookeeper for unit testing
Hi all, If someone is searching for some up-to-date examples of embedded kafka/zookeeper servers, like for unit testing, maybe will find this short blog post useful:...
View ArticleZkClient bug can bring down broker/consumer on zookeeper push in EC2 environment
Recently, we found the serious ZkClient bug, actual Apache Zookeeper client bug, which can bring down broker/consumer on zookeeper push. We're running kafka and zookeeeper in AWS EC2 environment....
View ArticleConsumer Parallelism
Hi, We are using the following method on ConsumerConnector to get multiple streams per topic, and we have multiple partitions per topic. It looks like only one of the runnable is active through a...
View Articlekafka TestUtils createBrokerConfig issue
Trying to write a unit test case for Kafka, and stuck with strange createBrokerConfig issue. on TestUtils.createBrokerConfigs(1) it gives me compilation error The method createBrokerConfigs(int,...
View Article