Design for Kafka
Hi, I’m new. We are considering using Kafka for our next generation product. We have 300K devices that are connected to our cloud backend. We are currently using a set of web services for our headless...
View ArticleCancel "kafka-reassign-partitions" Job
Hi All, First, I¹m using 0.8.1. Is there anyway to cancel an ongoing kafka-reassign-partitions job? I started a reassignment, and for some reason it just takes forever. However, it won¹t let me start...
View ArticleForcibly Move A Topic
Hi All, Is there anyway to move a topic by force, even if it means loss of messages? The problem I¹m looking at is the loss of multiple Kafka brokers simultaneously in a Kafka cluster, which means that...
View ArticleKafka 0.8's VerifyConsumerRebalance reports an error
Hi, I've run into the following problem. I try to read from a 50-partition Kafka topic using high level consumer with 8 streams. I'm using 8-thread pool, each thread handling one stream. After a short...
View ArticleHow does number of partitions affect sequential disk IO
I’ve been reading the Kafka docs and one thing that I’m having trouble understanding is how partitions affect sequential disk IO. One of the reasons Kafka is so fast is that you can do lots of...
View ArticleWhat's the behavior when Kafka is deleting messages and consumers are still...
When Kafka broker is trying to delete message according to the log retention setting, either triggered by log age or topic partition size, if the same time there are still consumers reading the topic,...
View ArticleAnnouncing Kafka Web Console v2.0.0
Announcing the second major release of Kafka Web Console: https://github.com/claudemamo/kafka-web-console/releases/tag/v2.0.0. Highlights: - I've borrowed some ideas from Kafka Offset Monitor and added...
View ArticleExperiences with larger message sizes
By any chance has anyone worked with using Kafka with message sizes that are approximately 50MB in size? Based on from some of the previous threads there are probably some concerns on memory pressure...
View ArticleApacha Kafka Commercial Support
Hello Are there any vendors who provide commercial support for Kafka? We're very interested in using kafka, but our infraestructure team ask us (DevTeam) for a commercial support. Tks diegoalvarez [ at...
View ArticleKafka 0.8/VIP/SSL
I'm looking for advice on running Kafka 0.8 behind VIPs. The goal is to support SSL traffic, with encryption and decryption being performed by back-to-back VIPs at the client and in front of the...
View ArticleUneven distribution of kafka topic partitions across multiple brokers
Have a kafka cluster with 10 brokers (kafka 0.8.0). All of the brokers were setup upfront. None was added later. Default number of partition is set to 4 and default replication to 2. Have 3 topics in...
View ArticleMonitoring Producers at Large Scale
We use Kafka as Transport Layer to transport application logs. How do we monitor Producers at large scales about 6000 boxes x 4 topic per box so roughly 24000 producers (spread across multiple data...
View ArticleBlacklisting Brokers
Hi All, Is there anyway to blacklist brokers? Sometimes we run into situations where there are certain hardware failures on a broker machine, and the machines goes into a “half dead” state. The broker...
View Articlekafka.common.LeaderNotAvailableException
Hi all, I have 3 zookeeper servers and 2 Kafka servers. Running Kafka version 0.8.1.1. Running zookeeper 3.3.5-cdh3u6. From the Kafka servers I can access the zookeeper servers on 2181. From one of the...
View ArticleGetting "java.io.IOException: Too many open files"
Hi All, I just upgraded my cluster from 0.8.1 to 0.8.1.1. I’m seeing the following error messages on the same 3 brokers once in a while: [2014-06-24 21:43:44,711] ERROR Error in acceptor...
View ArticleKafka connection loss with high volume of messages
Hello All, I am seeing this issue very frequently when running a high volume of messages through Kafka. It starts off well, and it can go on for minutes that way, but eventually it reaches a point...
View ArticleFailed to send messages after 3 tries
Hi, I get the following error from my producer when sending a message: Caused by: kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at...
View ArticleKafka partition reassignment script
Hi, I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like to add a new broker in the cluster. Each kafka instance has 400Go of data and we are using replication factor equals to 3...
View ArticleKafka 0.8.1.1 partition reassignment script
Hi, I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like to add a new broker in the cluster. Each kafka instance has 400Go of data and we are using replication factor equals to 3...
View ArticleReg Broker information.
Hi, Currently we have the command (/brokers/ids/0) to get the individual broker registration information. How do I get all the registered brokers from the zookeeper. Thanks Bala
View Article