Enforcing Network Bandwidth Quote with New Java Producer
HI Kafka Team, We like to enforce a network bandwidth quota limit per minute on producer side. How can I do this ? I need some way to count compressed bytes on producer ? I know there is callback does...
View Articleshrinking isr for partitions followed by error.. (kafka 0.8.1.1)
I had deleted the topic "sams_2014-10-30" successfully, and later on one of our kafka brokers (broker id 4) crashed, Now the topic is gone from zookeeper ls /brokers/topics (doesn't show the topic...
View Articlewill adding partitions to existing topic change leadership to existing...
Hi I want to add partitions to a running topic, and since I use the python producer I will eventually have to restart producers to reflect the change. the question is if leadership will change for the...
View Articleselecting java producer (0.8.2 or 0.8.1.1?)
Hi, I need to make a choice and I can't get a full picture on the differences between the two. E.g.: Are both producers async capable to the same extent? Is the new producer stable for production? Is...
View ArticleLong-delayed followup on weird topic corruption issue
Several months ago, I'd posted a lot of stuff here about an issue I was seeing, where the data segments for some topics were getting corrupted, for no obvious reason. We tried a bunch of stuff and got...
View ArticleISR shrink to 0?
We have had 2 nodes in a 4 node cluster die this weekend, sadly. Fortunately there was no critical data on these machines yet. The cluster is running 0.8.1.1, and using replication factor of 2 for 2...
View Articlekafka application logdir
How do I configure the application kafka log dir? Right now the default is /var/log/upstart/kafka.log . I want to point it to a different mount dir... e.g. /opt/kafka/bin/kafka-server-start.sh...
View ArticleAeron, a high throughput, low latency messaging system
Recently there has been some interesting discussion on the Mechanical Sympathy mailing list about Martin Thompson’s new extremely high performance, extremely low latency, open source messaging system...
View ArticleHow to recover from ConsumerRebalanceFailedException ?
Hi Kafka Team, I get following exception due to ZK/Network issues intermittently. How do I recover from consumer thread dying *programmatically* and restart source because we have alerts that due to...
View ArticleLarge Messages
Hi all, I am fairly new to Kafka and need some general advice on handling large messages. My message sizes range from 20k – 10M, with most of my messages upto 250K, with some messages ranging in sizes...
View ArticleConsumer and offset management support in 0.8.2 and 0.9
Hello everyone, I have a few questions about the current status and future of the Kafka consumers. We have been working to adding Kafka support in Spring XD [1], currently using the high level consumer...
View Articlebenchmark kafka on 10GbE network
Hi all, I have been trying out kafka benchmarks described in Jay's benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines...
View ArticleKafka consumer - want to print partition id along with message
Hi, I am trying to learn more on Kafka consuming patterns if we have partitions on the topic and testing few scenarios. I would like to print the message when the consumer consumes along with the...
View Articlepartition auto-rebalance
I'm trying to understand the config options for auto-rebalancing. This is what we have in /etc/kafka/server.properties for all the nodes: auto.leader.rebalance.enable=true...
View ArticleTopic Creation
Hi everyone, I'm looking for the API to create/manager topics. Is this class[1] the one I should be using? Also what if I'm wanting to create the topic synchronously? [1] -...
View ArticleUsing OffsetRequest to get the Head and Tail of a partition in a single request.
Hey all, We are working on a .net client, and I have a question about the OffsetRequest api...
View ArticleDrawbacks to setting log.roll.hours=1 ?
Hello, We have a 5 node cluster, each with a 5 disk JBOD storage running Kafka 0.8.1. We are interested in the possibility of using the OffsetRequest API taking a time to get offsets from a given time....
View Articlekafka message expiry question
hi, as far as i understand, log retention time in kafka will delete message that's older than the retention time. i'm wonder what it implies for consumer since i'm using simple consumer to manage...
View ArticleIssues with Kafka async producer's enqueue timeout
Hi, We are using an async producer to send data to a kafka cluster.The event rate at peak is around 250 events/second of size 25KB each. In the producer code base we have added specific debug...
View ArticleTTL changes - Are they retroactive?
Hi all, I had a question about TTL changes to a topic’s configuration. When I make the change and restart kafka, will kafka apply the changes to all existing messages on disk retroactively or would it...
View Article
