Quantcast
Channel: Kafka Timeline

Command to list my brokers

$
0
0
Hi all,

I just want a way to query all of my brokers to see if they're all
connected and online, without creating a topic. Or is creating a topic the
best way to verify all my brokers are up and running???

Thanks

Query regarding serialization

$
0
0
Hi,

I am new to kafka and still learning .I have a query .. As per my
understanding the serialization is happening before the partitioning and
grouping of messages per broker . Is my understanding correct and what is
the reason for the same?

Regards,
Liju John

In Apache Kafka, how can one achieve delay queue support (similar to what ActiveMQ has)?

$
0
0
Hi Kafka Users Community,
In Apache Kafka, how can one achieve delay queue support (similar to what ActiveMQ has)? Has anyone solved similar problem before? 
Thanks in advance?

Regards,Vish

kafka shutdown automatically

$
0
0
Hi,

our kafka cluster is shut down automatically today, here is the log.

I don't find any error log. Anything wrong?

[2015-01-18 05:01:01,788] INFO [BrokerChangeListener on Controller 0]:
Broker change listener fired for path /brokers/ids with children 0
(kafka.controller.ReplicaStateMachine$BrokerChangeListener)

[2015-01-18 05:01:01,791] INFO [Controller-0-to-broker-1-send-thread],
Shutting down (kafka.controller.RequestSendThread)

[2015-01-18 05:01:01,792] INFO [Controller-0-to-broker-1-send-thread],
Stopped (kafka.controller.RequestSendThread)

[2015-01-18 05:01:01,792] INFO [Controller-0-to-broker-1-send-thread],
Shutdown completed (kafka.controller.RequestSendThread)

[2015-01-18 05:01:01,792] INFO [Controller-0-to-broker-0-send-thread],
Shutting down (kafka.controller.RequestSendThread)

[2015-01-18 05:01:01,792] INFO [Controller-0-to-broker-0-send-thread],
Stopped (kafka.controller.RequestSendThread)

[2015-01-18 05:01:01,792] INFO [Controller-0-to-broker-0-send-thread],
Shutdown completed (kafka.controller.RequestSendThread)

[2015-01-18 05:01:01,792] INFO [Controller 0]: Controller shutdown complete
(kafka.controller.KafkaController)

can't iterate consumed messages when checking errorCode first

$
0
0
Hi all,

I'm using Kafka low level consumer api and find in the below codes
"iterator.hasNext" always return false. Through debugging, I'm sure the
messageSet has the size of "fetchSize"

* val consumer = new SimpleConsumer(broker.host, broker.port,
soTimeout, soBufferSize, clientId)*
* val request = new FetchRequestBuilder()*
* .addFetch(topic, partition, offset, fetchSize)*
* .build()*
* val response = consumer.fetch(request)*
* response.errorCode(topic, partition) match {*
* case NoError => {*
* iterator = response.messageSet(topic, partition).iterator*
* }*
* case error => throw exceptionFor(error)*
* }*

The weird thing is that the iterator works fine when I get iterator
directly without checking the error code.

* val consumer = new SimpleConsumer(broker.host, broker.port,
soTimeout, soBufferSize, clientId)*
* val request = new FetchRequestBuilder()*
* .addFetch(topic, partition, offset, fetchSize)*
* .build()*
* consumer.fetch(request).messageSet(topic, partition).iterator*

Any thoughts ?

Thanks,
Manu Zhang

Issue size message

$
0
0
Hi All,
I am having an issue when using kafka with librdkafka. I've changed the message.max.bytes to 2MB in my server.properties config file, that is the size of my message, when I run the command line ./rdkafka_performance -C -t test -p 0 -b computer49:9092, after consume some messages the consumer remain waiting something that don't arrive. My producer continues sending messages. Some idea?

% Using random seed 1421685059, verbosity level 1
% 214 messages and 1042835 bytes consumed in 20ms: 10518 msgs/s and 51.26 Mb/s, no compression
% 21788 messages and 106128192 bytes consumed in 1029ms: 21154 msgs/s and 103.04 Mb/s, no compression
% 43151 messages and 210185259 bytes consumed in 2030ms: 21252 msgs/s and 103.52 Mb/s, no compression
% 64512 messages and 314233575 bytes consumed in 3031ms: 21280 msgs/s and 103.66 Mb/s, no compression
% 86088 messages and 419328692 bytes consumed in 4039ms: 21313 msgs/s and 103.82 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 5719ms: 17571 msgs/s and 85.67 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 6720ms: 14955 msgs/s and 72.92 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 7720ms: 13018 msgs/s and 63.47 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 8720ms: 11524 msgs/s and 56.19 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 9720ms: 10339 msgs/s and 50.41 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 10721ms: 9374 msgs/s and 45.71 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 11721ms: 8574 msgs/s and 41.81 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 12721ms: 7900 msgs/s and 38.52 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 13721ms: 7324 msgs/s and 35.71 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 14721ms: 6826 msgs/s and 33.29 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 15722ms: 6392 msgs/s and 31.17 Mb/s, no compression
% 100504 messages and 490022646 bytes consumed in 16722ms: 6010 msgs/s and 29.30 Mb/s, no
........

The software when consume all offset send me the message:

% Consumer reached end of unibs.nec [0] message queue at offset 229790
RD_KAFKA_RESP_ERR__PARTITION_EOF: [-191]

However changed de message.max.bytes to 2MB I don’t receive the code from Kafka.

Anyone has some idea?

Thanks guys.

Jar files needed to run in Java environment (without Maven)

$
0
0
Hello All,

I posted a question last week (
http://mail-archives.apache.org/mod_mbox/kafka-users/201501.mbox/browser)
but I'm not able to respond to the thread for some reason?

The suggestion on the last post was that I am missing some jar files in my
classpath, but I am using the following jar files:

* kafka_2.10-0.8.2-beta.jar, kafka-clients-0.8.2-beta.jar,
metrics-core-2.2.0.jar, scala-library-2.10.4.jar, slf4j-api-1.5.6.jar,
slf4j-log4j12-1.5.6.jar*

These are the jar files that I used when I ran the Kafka producer using the
Spark submit script which looked like this:

*/opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/spark/bin/spark-submit
--class SimpleApp --local[2] --jars $(echo
/home/ec2-user/sparkApps/lib/*.jar | tr ' ' ',')
/home/ec2-user/target/simple-project-1.0.jar*

Is there a reason I would need additional jar files now that i'm using the
java -cp -jar command?

Thanks!

Join


Kafka Out of Memory error

$
0
0
Hi All,

I have a kafka cluster setup which has 2 topics

topic1 with 10 partitions
topic2 with 1000 partitions.

While, I am able to consume messages from topic1 just fine, I get following
error from the topic2. There is a resolved issue here on the same thing
https://issues.apache.org/jira/browse/KAFKA-664

I am using latest kafka server version, and I have used kafka command line
tools to consumer from the topics.

[2015-01-19 22:08:10,758] ERROR OOME with size 201332748
(kafka.network.BoundedByteBufferReceive)

java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
at
kafka.network.BoundedByteBufferReceive.byteBufferAllocate(BoundedByteBufferReceive.scala:80)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:63)
at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:81)
at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:107)
at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
<<<

Thanks
-Pranay

0.8.2.0 Candidate 1

$
0
0
Thanks for reporting the issues in RC1. I will prepare RC2 and start a new
vote.

Jun

java.nio.channels.ClosedChannelException...Firewall Issue?

$
0
0
Hello Everyone,

Thank you for the help!

Preface: I've created producers/consumers before and they have worked. I
have also made consumers/producers using java programs, but they have all
been locally.

1) I have a Zookeeper/Kafka Server running on an EC2 instance called "A"

2) I started the Zookeeper/Kafka Server on A and created a topic "test"
like it says on the kafka documentation

3) I then started a console consumer on A like the documentation states for
test.

4) I then downloaded Kafka on EC2 instance called "B" and created a console
producer with broker with broker-list public.ip.of.A:9092 and topic test.

5) Once I start publishing messages this is the message I get (what's worse
is that I can't cntrl C to stop and have to exit putty):

WARN Fetching topic metadata with correlation id 16 for topics [Set(test)]
from broker [id:0,host:54.183.40.224,port:9092] failed
(kafka.client.ClientUtils$)

java.nio.channels.ClosedChannelException

at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)

at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)

at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:73)

at kafka.producer.SyncProducer.send(SyncProducer.scala:114)

at
kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)

at
kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)

at
kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)

at kafka.utils.Utils$.swallow(Utils.scala:172)

at kafka.utils.Logging$class.swallowError(Logging.scala:106)

at kafka.utils.Utils$.swallowError(Utils.scala:45)

at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)

at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)

at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:93)

at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)

[2015-01-20 01:29:28,289] ERROR fetching topic metadata for topics
[Set(test)] from broker [ArrayBuffer(id:0,host:

Meanwhile in ZK...not sure if this happened as soon as I started publishing
messages:

Error:KeeperErrorCode = NoNode for /consumers/console-consumer-2615/offsets
(org.apache.zookeeper.server.PrepRequestProcessor)

Any tips/suggestions are greatly appreciated!

0.8.2.0 Candidate 2

$
0
0
This is the second candidate for release of Apache Kafka 0.8.2.0. There has
been some changes since the 0.8.2 beta release, especially in the new java
producer api and jmx mbean names. It would be great if people can test this
out thoroughly.

Release Notes for the 0.8.2.0 release
*https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/RELEASE_NOTES.html
<https://people.apache.org/~junrao/kafka-0.8.2.0-candidate1/RELEASE_NOTES.html>*

*** Please download, test and vote by Monday, Jan 26h, 7pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
*http://kafka.apache.org/KEYS <http://kafka.apache.org/KEYS>* in addition
to the md5, sha1 and sha2 (SHA256) checksum.

* Release artifacts to be voted upon (source and binary):
*https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/
<https://people.apache.org/~junrao/kafka-0.8.2.0-candidate1/>*

* Maven artifacts to be voted upon prior to release:
*https://repository.apache.org/content/groups/staging/
<https://repository.apache.org/content/groups/staging/>*

* scala-doc
*https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/scaladoc/
<https://people.apache.org/~junrao/kafka-0.8.2.0-candidate1/scaladoc/#package>*

* java-doc
*https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/javadoc/
<https://people.apache.org/~junrao/kafka-0.8.2.0-candidate1/javadoc/>*

* The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.0 tag
*https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=058d58adef2ab2787e49d8efeefd61bb3d32f99c
<https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=058d58adef2ab2787e49d8efeefd61bb3d32f99c>*
(commit 0b312a6b9f0833d38eec434bfff4c647c1814564)

/*******************************************

Thanks,

Jun

How to setup inter DC replication in Kafka 0.8.1.1

$
0
0
Hi,

I want to setup inter DC replication b/w Kafka clusters. Is there any
inbuilt tool to do this?

I already have tried MirrorMaker tool but the problem is, if MM killed then
some messages get duplicated. I don't want to duplicate the messages.

Please suggest a way to do this. Please share your experience how you have
setup this.

Thanks in Advance,
Madhukar

How to handle broker disk failure

$
0
0
I'm trying to figure out the best way to handle a disk failure in a live
environment.

The obvious (and naive) solution is to decommission the broker and let
other brokers taker over and create new followers. Then replace the disk
and clean the remaining log directories and add the broker again.

The disadvantage with this approach is of course the network overhead and
the time it takes to reassign partitions.

Is there a better way?

As a sub question, is it possible to continue running a broker with a
failed drive and still serve the remaining partitions?

thanks,
svante

typo in wiki

$
0
0
In the wiki - there is a statement that a partition must fit on a single
machine, while technically true, isn't it so that a partition must fit on a
single disk on that machine.

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowmanytopicscanIhave

partition in your topic you >cannot scale your write rate or retention
beyond the capability of a single machine. If you have 1000 >partitions you
could potentially use 1000 machines.

thanks,
svante

Kafka-Storm Integration

$
0
0
Hi,

I am runnning a Kafka-storm topology .

The topology works exactly as expected in LocalCluster mode. However in
Distributed mode the Workers are not starting , supervisor has not started.
Only nimbus starts

I am trying to run the topology via eclipse.

//Providing the details for zkHosts
BrokerHosts hosts = new ZkHosts("hbase:2181","/brokers");
SpoutConfig spoutconf = new SpoutConfig(hosts,TOPIC_NAME,"/test101","sid");
spoutconf.scheme = new SchemeAsMultiScheme(new StringScheme());
spoutconf.forceFromStart = true;
KafkaSpout Kspout = new KafkaSpout(spoutconf);
builder.setSpout("spout", Kspout, 1);

PhoenixBolt phoenixBolt = new PhoenixBolt("Phoenix-Bolt", true,true,new
Fields("json"));
builder.setBolt("Phoenix-Bolt", phoenixBolt, 1).shuffleGrouping("spout");

Config config = new Config();

config.put(Config.NIMBUS_THRIFT_PORT,6627);
config.put(Config.STORM_ZOOKEEPER_PORT,2181);
config.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new
String[]{"hbase"}));
config.put(Config.NIMBUS_HOST, "dwh3");
config.put("storm.zookeeper.root","/storm1");
config.put(Config.STORM_ZOOKEEPER_ROOT, "/storm1");

System.setProperty("storm.jar","/usr/local/apache-storm-0.9.3/bin/storm");
config.setDebug(true);
config.setNumWorkers(5);
config.put(Config.TOPOLOGY_DEBUG, true);
System.out.println(config.toString());

//cluster.submitTopology("kafka-storm-Phoenix", config,
//builder.createTopology());
StormSubmitter.submitTopology("kafka-storm-Phoenix-ClusterD", config,
builder.createTopology());

worker logs are saying " Netty reconnect for hostname:6703" is issued again
and again.

Thanks,
Siddharth

Updated Logstash-Kafka plugin

$
0
0
Hello Everyone,

As we wait for the 1.5 release of Logstash which will include the Kafka plugin by default I've put some work into the logstash-kafka plugin which makes it much easier to install and also enables logging on from the kafka library when you use --debug or --verbose toggles with logstash. The jruby-kafka library underneath now includes all jars needed to run jruby-kafka so no more manual Kafka install! Yay! Finally, I have packaged up logstash-kafka into a gem and distributed it via rubygems to make for really easy installation. So yay again! Check it out here: https://github.com/joekiller/logstash-kafka#rubygems-logstash-kafka-install.

Regards,

Joe Lawson

How to run Kafka Producer in Java environment? How to set mainClass in pom file in EC2 instance?

$
0
0
Hello Everyone,

Sorry for the duplicate post, I hadn't subscribed to the list yet, so I
couldn't reply to suggestions. That has been fixed now.

1) Using Maven, I wrote a Kafka Producer similar to the one found here:
https://github.com/pppsunil/HelloKafka/blob/master/src/main/java/com/spnotes/kafka/HelloKafkaProducer.java

2) This is my pom file:

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="
http://maven.apache.org/POM/4.0.0" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion> <groupId>HelloKafka</groupId>
<artifactId>HelloKafka</artifactId> <version>1.0-SNAPSHOT</version> <build>
<plugins> <plugin> <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId> <version>2.2</version> <!--
nothing here --> </plugin> <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-4</version> <configuration> <descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs>
<archive> <manifest> <mainClass>HelloKafkaProducer</mainClass> </manifest>
</archive> </configuration> <executions> <execution> <phase>package</phase>
<goals> <goal>single</goal> </goals> </execution> </executions> </plugin>
</plugins> </build> <dependencies> <dependency>
<groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10</artifactId>
<version>0.8.2-beta</version> <scope>compile</scope> <exclusions>
<exclusion> <artifactId>jmxri</artifactId> <groupId>com.sun.jmx</groupId>
</exclusion> <exclusion> <artifactId>jms</artifactId>
<groupId>javax.jms</groupId> </exclusion> <exclusion>
<artifactId>jmxtools</artifactId> <groupId>com.sun.jdmk</groupId>
</exclusion> </exclusions> </dependency> <dependency>
<groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId>
<version>1.5.6</version> </dependency> <dependency>
<groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId>
<version>2.10.4</version> </dependency> <dependency>
<groupId>org.scala-lang</groupId> <artifactId>scala-compiler</artifactId>
<version>2.10.4</version> </dependency> <dependency>
<groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10</artifactId>
<version>0.8.2-beta</version> </dependency> <dependency>
<groupId>com.101tec</groupId> <artifactId>zkclient</artifactId>
<version>0.3</version> </dependency> <dependency>
<groupId>com.yammer.metrics</groupId> <artifactId>metrics-core</artifactId>
<version>2.2.0</version> </dependency> <dependency>
<groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId>
<version>0.8.2-beta</version> </dependency> <dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId> <version>2.4</version>
</dependency> </dependencies> </project>

3) I am really lost on how to run kafka producer from the command line. I
have followed the instructions here as seen in my above pom:
http://stackoverflow.com/questions/1814526/problem-building-executable-jar-with-maven

4) So I have my HKP.java file, then the pom file I have shown above and
then I do mvn clean install. It builds the target directory with two jar
files...what is my next step? I try java -cp jar.with.dependencies.jar -jar
main.jar, but it says no main manifest attribute. I try java -cp
jar.with.dependencies HKP, but it says it can't find class HKP

I have a feeling I am not writing the mainClass properly as most online
examples say com.example.class, but I'm not sure how that translates to
working on EC2.

Thanks a lot for the help!

-Su

Poll: Producer/Consumer impl/language you use?

$
0
0
Hi,

I was wondering which implementations/languages people use for their Kafka
Producer/Consumers.... not everyone is using the Java APIs. So here's a
1-question poll:

http://blog.sematext.com/2015/01/20/kafka-poll-producer-consumer-client/

Will share the results in about a week when we have enough votes.

Thanks!
Otis

Can I run Kafka Producer in BG and then run a program that outputs to the console?

$
0
0
Hello Everyone,

Sorry for asking multiple questions, but I am currently trying another
approach to run a kafka producer.

1) I started the kafka console producer as mentioned here in the background
(just added a to the end of the producer script) :
http://kafka.apache.org/documentation.html#introduction

2) I then ran a Java script that publishes messages to the console, like
Hello World, etc, but the messages did not get published (I know the
connection was set up as if I manually typed in messages, I was able to see
them in my consumer).

3) Is this set up possible?

Thanks!

-Su




Latest Images