Recall: Socket is not connected error while consuming messages using kafka
Pradeep Simha (WT01 - Business Outcome Services) would like to recall the message, "Socket is not connected error while consuming messages using kafka". The information contained in this electronic...
View ArticleInformal Kafka logstash user survey
Hi everyone, The maintainers of logstash is considering adding my Kafka plugin (https://github.com/joekiller/logstash-kafka ) to the core of logstash. They are asking at the following link for some...
View ArticleAbility to Inject Queue Implementation Async Mode
Kafka Version: 0.8.x 1) Ability to define which messages get drooped (least recently instead of most recent in queue) 2) Try Unbounded Queue to find out the Upper Limit without drooping any messages...
View ArticleUniform Distribution of Messages for Topic Across Partitions Without...
How to achieve uniform distribution of non-keyed messages per topic across all partitions? We have tried to do this uniform distribution across partition using custom partitioning from each producer...
View ArticleConflict stored data in Zookeeper
Hi, everyone. I'm using 0.8.1.1, and I have 8 brokers and 3 topics each have 16 partitions and 3 replicas. I got unseen logs like below. this is occur every 5 seconds. [2014-08-05 11:11:32,478] INFO...
View Articlehigh level consumer api blocked forever
Hi, every one. I got into a strange case that my consumer using high level api worked fine at first, but couple days later blocked in ConsumerIterator.hasNext(), while there are pending messages on the...
View ArticleConsumer is never shutdown
Hi, I just started with Apache Kafka and wrote a high level consumer program following the example given here https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example. Though, I was...
View ArticleIs there a way to stop/pause producers from writing?
Is there a tool or way using the Kafka admin api to tell a producer to stop/pause writing to a specific topic? The use case is basically we need to stop writing to a topic, let the consumers get caught...
View Articlekafka.SocketServerStats write rate metrics seem wrong
Hi. I'm seeing some odd numbers from kafka.SocketServerStats. Ideally, I'd like to have stats broken down per-topic, e.g. what's our most written/read topics? For write rates, I've got a separate...
View ArticleApache webserver access logs + Kafka producer
Hi, I want to collect apache web server logs in real time and send it to Kafka server. Is there any existing Producer available to do this operation, If not can you please provide a way to implement...
View Articletrying to tune kafka's internal logging - need help...
Hi, I am trying to get rid of the log files written under “$base_dir/logs”, folder create by line 26 at “bin/kafka-run-class.sh”. I use an EC2 machine with small primary disk and it blows away on...
View ArticleIssue with kafka-server-start.sh
Hi, I am working with kafka_2.8.0-0.8.1.1. It used to work fine but since this morning when I try to start the kafka-server after starting zookeeper, I am getting this error- INFO I wrote this...
View Articleconsumer rebalance weirdness
We've noticed that some of our consumers are more likely to repeatedly trigger rebalancing when the app is consuming messages more slowly (e.g. persisting data to back-end systems, etc.). If on the...
View Articleerror recovery in multiple thread reading from Kafka with HighLevel api
Folks, I have a process started at specific time and read from a specific topic. I am currently using the High Level API(consumer group) to read from kafka(and will stop once there is nothing in the...
View ArticleArchitecture: amount of partitions
Dear all, I'm new to Kafka, and I'm considering using it for a maybe not very usual purpose. I want it to be a backend for data synchronization between a magnitude of devices, which are not always...
View ArticleA wired producer connection timeout issue
A Kafka producer frequently timeout when connecting to a remote Kafka cluster while producers on other machine (same data center) can connect to the Kafka cluster with no problem. From the monitoring,...
View ArticleError while producing messages
Hi, I am trying to do capacity sizing estimate for our kafka cluster. I started with 5 broker cluster and 3 node zk. Used a simple java based producer to send messages to 5 topics that are created in...
View Articlegetting socket timeout
Hi Team, I am bit new to kafka. I was trying to set up kafka 0.8 and connect it to druid firehose. My console producer is working fine but when I try to connect to it using java code I am getting...
View ArticleKafka Consumer not consuming in webMethods.
Hi Jun/Neha/Team, We are trying to consume from Kafka using webMethods as our consumer. When we start the consumer, fetcher and leader threads went into WAITING state and not consuming anything, when I...
View ArticleLeaderNotAvailableException
I have a single broker test Kafka instance that was running fine on Friday (basically out of the box configuration with 2 partitions), now I come back on Monday and producers are unable to send...
View Article