kafka TestUtils createBrokerConfig issue
Trying to write a unit test case for Kafka, and stuck with strange createBrokerConfig issue. on TestUtils.createBrokerConfigs(1) it gives me compilation error The method createBrokerConfigs(int,...
View Articlefsync behaviour
Hi, I wanted to understand the exact behaviour of kafka ( 0.8.1.1 ) w.r.t fsync. I have read the documentation and some related forum posts, but wanted to confirm the understanding here. 1) If we set...
View Articlekafka TestUtils choosePort NoSuchMethodError
I am trying to create unit test case for Kafka with a simple call Properties props =TestUtils.createBrokerConfig(1, TestUtils.choosePort(), true); It fails on I also tried:...
View ArticleData inputs for Kafka.
Hi guys, We are implementing a messaging system in order to perform Real time analytics. However, we are confused on using ActiveMQ or Apache Kafka. The point of worry for us are : 1. What kind of data...
View ArticleWhich is better?
Hi All, My requirement is to consume data from kafka using consumer. Which should be better to use? SimpleConsumer or HighLevel Consumer? Thanks, Pankaj Ojha
View ArticleAnnounce: Capillary, a monitor for Kafka 0.8 spout topologies
Hello all! If you use Storm and Kafka 0.8 together I may have an interesting project for you. Recently at Keen IO <https://keen.io/> we upgraded from Kafka 0.7 to 0.8 and needed to replace the...
View ArticleKafka Mirroring Issue
Hello All, When i start the mirroring process, events are written only to partition "1" on the target cluster and no data is written to partition 0. I am stuck on this and wondering what am i doing...
View Articledoubt regarding High level consumer commiting offset
Hi, I have My Kafka setup with 0.8.1 kafka with 3 producers and 3 consumers. I use the high level consumer. My doubt is , i read the messages from the consumer iterator, and after reading the message i...
View Articleread message use SimpleConsumer,can not start with assign offset
kafka version is 0.8.0, the code as https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example when i assign a offset ,it will throw java.lang.ArrayIndexOutOfBoundsException: 0...
View ArticleKafka build for Scala 2.11
Hi Team, We are trying to compile 0.8.1.1 with Scala 2.11 and its giving me compilation errors. Please let me know which patch should I apply from below JIRA.I tried with latest one and it failed to...
View ArticlePlease fix broken link and update blurb
Could someone with edit rights in the wiki please fix the broken link for Metamarkets that's on this page: https://cwiki.apache.org/confluence/display/KAFKA/Powered+By The right URL is:...
View Articlerestoring broker node without Zookeeper data?
Running isolated primary, secondary, etc, Kafka brokers without replication (clients failover to other Kafka brokers if needed). Each broker instance has a single local Zookeeper instance. Using...
View Articlestep by step guide for writing a non-java 0.9 client
The 0.9 Consumer re-write design document refers to a "step by step guide for writing a non-java 0.9 client" but the link doesnt work -- it appears to be an internal LinkedIn document? Is this...
View Articlekafka unit test in Java, TestUtils choosePort sends NoSuchMethodError
I am trying to create unit test case for Kafka in Java with a simple call Properties props =TestUtils.createBrokerConfig(1, TestUtils.choosePort(), true); It fails on I also tried:...
View ArticleOutOfMemoryError during index mapping
Hi, Sometimes we get: Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885) ... 25 more (excuse...
View ArticleHow retention is working
Hi! I'm wondering how the retention time is working exactly... I know that log.retention.{minutes,hours} and retention.ms (by topic) to set the retention time and that there is...
View ArticleEBCDIC support
Hey all, This might seem like a silly question, but does kafka have support for EBCDIC? Say I had to read data from an IBM mainframe via a TCP/IP socket where the data resides in EBCDIC format, can...
View ArticleMigrating data from old brokers to new borkers question
Hi, I would like to migrate my Kafka setup from old servers to new servers. Let say I have 8 really old servers that have the kafka topics/partitions replicated 4 ways and want to migrate the data to 4...
View ArticleKafka Log Cleaner Stopped (?)
Hi, We're using compaction on some of our topics. The log cleaner output showed that it kicked in when the broker was restarted. But now after several months of uptime, the log cleaner output is empty....
View ArticleMore partitions than consumers
Hi, As far as I can see, the (otherwise great and very helpful) documentation isn't explicit about this, but: given more partitions than consumers, will all messages still be read? I've discussed this...
View Article