Running on Windows in production
Hi, I am looking into using Kafka for production to store notifications sent by one of our services. The service clients will then be able to either replay lost notifications while they were down or...
View Articlecount of bytes stored in kafka
Hi All, Can we see count of messages or bytes of a topic present in kafka using jconsole or any other way ? Jmx metric "BytesInperSec" doesn't show data stored in kafka. As per my understanding:...
View ArticleConsumerRebalanceFailedException
Hi All, I am getting consumer rebalance failed exception if i restart my consumer within 1-3 seconds. Exception trace is: Caused by: kafka.common.ConsumerRebalanceFailedException:...
View ArticleManual Partition-wise Offset Management
Hi All Can we manage partition-offset commits separately instead of a consumerConnector.commitOffsets() (commits offsets of all the related broker partitions simultaneously) call ?
View ArticleLarge number of partitions
I'm looking to create a topic with about 1400 partitions to allow a high degree of parallel processing. We have 5 brokers so that would be 280 partitions per box. Has anyone done something with this...
View Articleonly one ProducerSendThread thread when running with multiple brokers (kafka...
Hi, I am using kafka 0.8. I have 3 machines each running kafka broker. I am using async mode of my Producer. I expected to see 3 different threads with names starting with ProducerSendThread-...
View Articlejava.lang.IllegalArgumentException Buffer.limit on FetchResponse.scala + 33
While consuming from the topics I get an IlegalArgumentException and all consumption stops, the error keeps on throwing. I've tracked it down to FectchResponse.scala line 33 The error happens when the...
View Articlekafka + storm
I have briefly looked at storm, but just a quick question, storm seems to have all these workers but they way it seems to me the order in which these items are processed off the queue is very random...
View Articlewhich zookeeper version
Hi, I am starting a fresh deployment of kafka + zookeeper.Looking at zookeeper releases find 3.4.5 old and stable enough.Has anyone used this before in production? kafka ops wiki page says at Linkedin...
View ArticleMeasuring remaining data
I need to provide a report that shows the total amount of unread data available to a consumer group for a particular set of topics. Right now I do this by adding custom offset tracking logic to the...
View ArticleTrouble recovering after a crashed broker
Hi all, We have a cluster of 3 0.8 brokers, and this morning one of the broker crashed. It is a test broker, and we stored the logs in /tmp/kafka-logs. All topics in use are replicated on the three...
View ArticleFormat of Kafka storage on disk
Is there any place where I can know about the internal structure of the log file where kafka stores the data. A topic has a .index and a .log file. I want to read the entire log file and parse the...
View ArticleConsumer not consuming.
Consumer offset checker shows its connected to consumer(account-access-hadoop-consumer_tm1mwdpl03-1383065261413-15d3cb41-0) but that consumer is not started,what could be reason its showing its...
View ArticleIs this the right official logo for Apache Kafka?
http://upload.wikimedia.org/wikipedia/commons/f/f7/Apache_kafka.png Please help me confirm. Thanks, Menka "Don't let what you cannot do interfere with what you can do" - by John Wooden
View ArticleWhy does the high level consumer block, or rather where does it?
I'm trying to trace through the codebase and figure out where exactly the block occurs in the high level consumer? public void run() { ConsumerIterator<byte[], byte[]> it = m_stream.iterator();...
View Articlesingle node to multiple broker cluster help
Hi, I'm working through some examples and I'm having issues with activating more than one server. I'm running on an AWS instance on an Amazon Linux AMI. The single node to single broker worked like a...
View Articlekafka.common.NoBrokersForPartitionException: Partition = null
Hi, all ~ There are 3 zookepers and one broker, Kafka 0.7.2. I got some temporary problems in zk cluster enviornment, but, any zk processes didn't go down. A broker got zk session timeout message, then...
View ArticlePartitionOffsetRequestInfo only supports -1 and -2?
Does it support an long timestamp? Tried it and the resulting offset response contains an empty long array.
View Article0.8 high-level consumer error handling
Hi - I noticed that if a kafka cluster goes away entirely, the high-level consumer will endlessly try to fetch metadata until the cluster comes back up, never bubbling the error condition up to the...
View Articlekafka-s3-consumer
Will the current kafka-s3-consumer ( https://github.com/razvan/kafka-s3-consumer) work with 0.8.0? Ray Rodriguez Medidata Solutions
View Article