Question on Kafka .8 / Logstash client
Hi Kafka Users, 1. Is there a plugin/client for Logstash for Kakfa 0.8 ? 2. Is there any example of using Kafka using Logstash? I'm not in the mailing list yet so I'll really appreciate a email...
View ArticleRecovering from consumer failure
I have a consumer program that encountered an error when handling a message and crashed. It does manual commits, and since it failed, it keeps failing now because it gets the same bad message. I ran...
View Articlere-writing old fetch request to work with 0.8 version
I found this embedded kafka example online ( https://gist.github.com/mardambey/2650743) which I am re-writing to work with 0.8 Can someone help me re-write this portion: val cons = new...
View ArticleReliable Message Commits
I'm using Kafka 0.8.1.1. I have a simple goal: use the high-level consumer to consume a message from Kafka, publish the message to a different system, and then commit the message in Kafka. Based on my...
View ArticleHelp is processing huge data through Kafka-storm cluster
Hi, Daily we are downloaded 28 Million of messages and Monthly it goes up to 800+ million. We want to process this amount of data through our kafka and storm cluster and would like to store in HBase...
View Articleusername related weirdness with kafka 0.8.1.1 (scala 2.10.1)
hi. as subj. states, I'm trying to use kafka-0.8.1.1 with samza (0.7). Initially the work started as a dev. user, e.g. "devuser", and kafka would run as that user, "and it's been good". Then, during...
View ArticleUse Kafka To Send Files
Hi Sometimes we want use kafka to send files(like textfile,xml...), but I didn't see this in documention. Can kafka use to tansfer files? If can ,how can I do Thanks
View Articledata archiving
Hi all, I'm looking for the way of archiving data. The data is hot for few days in our system. After that it can rarely be used. Speed is not so important for archive. Lets say we have kafka cluster...
View Articlekafka producer, one per web app?
In my web application, I should be creating a single instance of a producer correct? So in scala I should be doing something like: object KafkaProducer { // props... val producer = new Producer[AnyRef,...
View Articlelinkedin and pageview producer + when kafka is down
I'd love to get some insights on how things work at linkedin in terms of your web servers and kafka producers. You guys probably connect to multiple kafka clusters, so let's assume you are only...
View ArticleBroken download link?
Hi there, as I was excitedly reading through the introductory documentation, I came across a download link [1] in the quickstart section [2]. It seems like the file isn't there anymore. It didn't take...
View ArticleNewly added partitions in 0.8.1.1 dont see data immediately
Hi, I used the add partition functionality in create-topics to alter a previous topic and increase the partitions. I noticed that after the new partitions were added, they dont receive data immediately...
View Articlemirrormaker's configuration to minimize/prevent data loss
As I read, consumer and producer in mirrormaker are independent and use queue to communicate. Therefore consumers keep on consuming/commiting offsets to zk even if producer is failing. Is it still the...
View ArticleISR not updating
Hi, team. Im using Kafka 0.8.1.1. I'm running 8 brokers on 4 machine. (2 brokers on 1 machine) and I have 3 topics each have 16 partitions and 3 replicas. kafka-topics describe is Topic:topicCDR...
View ArticleUnit testing with Kafka
Are there unit testing libs in kafka which we can include to test our producers/consumers?? I found the following but the maven libs mentioned there seem to be missing....
View ArticleBuilding Kafka on Mac OS X
Hi team, I’m a newcomer to Kafka, but I’m having some troubles trying to get it to run on OS X. Basically building Kafka on OS X with 'gradlew jar’ gets stuck forever without any progress (Indeed I...
View Articlecannot replicate topics kafka inconsistent state
Hi, I've installed kafka 2.8.1, created a topic using: /opt/kafka/bin/kafka-topics.sh --create --topic "test" --zookeeper "localhost:2381" --partitions 2 --replication-factor 2 Then opened a console...
View ArticleOffset of last un-consumed message
Hi, I have a consumer group with multiple threads (high-level consumers) which read from a topic. I am also using a SimpleConsumer to read messages given a start offset. I am getting the offset as the...
View ArticleSupport for Java 1.8?
Hi, I was wondering if Kafka has been tested/certified under JRE 1.8? Thanks! Chris
View ArticleSteps to run Kafka 0.8.1 on Windows using cmd prompt ?
Need help to run kafka 0.8.1 on windows 7. Currently kafka 0.8.1 can't be run on windows using command prompt. I have gone through below post which have steps to run kafka 0.8.0....
View Article