ConsumerRebalanceFailedException

July 15, 2013, 10:15 am

≫ Next: Consumer offset oscillates between 2 values

≪ Previous: Protocol:Offsetrespose

$

0

0

Hi all,

We have a small Kafka cluster (0.7.1 - 3 nodes) in EC2. The load is about
200 million events per day, each being few kilobytes. We have a single node
zookeeper.

Yesterday suddenly our Kafka clients started throwing the following
exception:
java.lang.RuntimeException: kafka.common.ConsumerRebalanceFailedException:
CONSUMER_GROUP_NAME_ip-00-00-00-00.ec2.internal-1373821190828-5f78e9af
can't rebalance after 4 retries
at
com.gumgum.kafka.consumer.KafkaTemplate.executeWithBatch(KafkaTemplate.java:59)
at
com.gumgum.storm.fileupload.GenericKafkaSpout.nextTuple(GenericKafkaSpout.java:73)
at
backtype.storm.daemon.executor$fn__3968$fn__4009$fn__4010.invoke(executor.clj:433)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)

None of the Kafka clients (ConsumerConenctor class) would start. They would
fail with the exception.

We tried restarting the clilents, restarting the zookeeper as well. But
finally it all started working when we restarted all of our kafka brokers.
We didn't lose any data because producers (going directly to the brokers
through a load balancer) were working fine.

I tried googling this issue and looks like lot of people have faced it, but
couldn't get anything concrete.

Given this, I have two questions:

It will be nice if you can tell me why this can happen or point me to a
link where I can understand it better. What does Consumer Rebalancing mean?
Does that mean consumers are trying to coordinate amongst themselves using
Zookeeper?

On a separate note, are there any JMX parameters I need to be monitoring to
make sure that my kafka cluster is healthy? How can I keep watch on my
kafka cluster?

Regards,
Vaibhav Puranik
GumGum

---

Consumer offset oscillates between 2 values

July 15, 2013, 12:27 pm

≫ Next: Leader not local for partition error?

≪ Previous: ConsumerRebalanceFailedException

$

0

0

Hi,
I use a Java high level consumer to read from a topic with compression set
to snappy. When I query the consumer offset for that group using zookeeper
shell, the offset value oscillates between 2 values. I use kafka 0.8 and I
never faced this issue before. This happens even when there are no high
level consumers running. Could you please tell why this happens?

get /consumers/g1/offsets/z1/0
*45212345*
cZxid = 0x54a5aa
ctime = Mon Jul 08 14:10:00 EDT 2013
mZxid = 0x6bb643
mtime = Mon Jul 15 15:24:47 EDT 2013
pZxid = 0x54a5aa
cversion = 0
dataVersion = 857118
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 8
numChildren = 0

get /consumers/g1/offsets/z1/0
*28997714*
cZxid = 0x54a5aa
ctime = Mon Jul 08 14:10:00 EDT 2013
mZxid = 0x6bb67e
mtime = Mon Jul 15 15:25:07 EDT 2013
pZxid = 0x54a5aa
cversion = 0
dataVersion = 857157
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 8
numChildren = 0

Thanks
Arathi

Leader not local for partition error?

July 15, 2013, 1:36 pm

≫ Next: Sample Kafka Producer/Consumer example in Scala

≪ Previous: Consumer offset oscillates between 2 values

$

0

0

Hi all,

I'm not sure if I'm doing something wrong or if I missed a step
somewhere. A little while ago I successfully got the 0.8 quickstart
example to work fine with the console producer/consumer. Then I went to
work on some code to learn how to implement a producer, which failed
with the producer not being able to send anything with the following
error in the logs:

Produce request with correlation id 11 failed due to [test,0]: kafka.common.NotLeaderForPartitionException

So I went back to trying the console producer, and I'm getting the same
error. To be sure, I removed all generated data by ZooKeeper and Kafka
and re-followed the steps of the quickstart guide, but I'm getting the
same error with the console producer/consumer.

kafka-list-topic.sh correctly lists my 1-partition, 1-replica test
topic:

topic: test partition: 0 leader: 0 replicas: 0 isr: 0

ZK and the broker are of course both up and running. Starting the
producer nothing out of the ordinary happens, but when starting the
consumer (before attempting to send anything), I get the following
exception:

[2013-07-15 13:25:46,487] INFO [ConsumerFetcherThread-console-consumer-943_polygon-1373919946074-f478ba53-0-0], Starting (kafka.consumer.ConsumerFetcherThread)
[2013-07-15 13:25:46,517] ERROR [console-consumer-943_polygon-1373919946074-f478ba53-leader-finder-thread], Error due to (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.NotLeaderForPartitionException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

I'm at a loss at what's going on here. When the broker starts it clearly
goes through the election process and becomes the leader (since it's the
only broker anyway) for the 'test' topic:

[2013-07-15 13:33:05,345] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2013-07-15 13:33:05,550] INFO [Replica Manager on Broker 0]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:3;CorrelationId:0;ClientId:id_0-host_null-port_9092;PartitionState:(test,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:polygon.turn.com,port:9092 (kafka.server.ReplicaManager)
[2013-07-15 13:33:05,551] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2013-07-15 13:33:05,563] INFO [Kafka Server 0], Started (kafka.server.KafkaServer)
[2013-07-15 13:33:05,563] INFO [ReplicaFetcherManager on broker 0] Removing fetcher for partition [test,0] (kafka.server.ReplicaFetcherManager)
[2013-07-15 13:33:05,566] INFO [Replica Manager on Broker 0]: Handled leader and isr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:3;CorrelationId:0;ClientId:id_0-host_null-port_9092;PartitionState:(test,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:polygon.turn.com,port:9092 (kafka.server.ReplicaManager)

I'm running Kafka from branch 0.8 (b1891e7). Any idea what's going on there?

Thanks,
/Max

Sample Kafka Producer/Consumer example in Scala

July 16, 2013, 9:21 am

≫ Next: Topic corruption from hardware failure (0.7.1)

≪ Previous: Leader not local for partition error?

$

0

0

Hello All

I have been attempting to build an end-to-end example of producer with
multiple consumers using scala.
I managed to build a producer, but when I attempt to build a consumer the
following statement causes the scala REPL to block/hang

val consumerMap = consumer.createMessageStreams(topicCountMap)

Am I missing anything here?

Can anybody please point me to a simple producer with multiple consumers
example in scala?

*Please do not point me to ConsumerConsumer or MirrorMaker as both of them
are fairly complex to even look at (given that I am a novice scala
programmer)*

Any help will be appreciated?

Thanks in advance
Amar

Topic corruption from hardware failure (0.7.1)

July 16, 2013, 1:10 pm

≫ Next: Message Encoding in Kafka 0.8

≪ Previous: Sample Kafka Producer/Consumer example in Scala

$

0

0

Hi everyone,

Last week, one of our production Kafka 0.7.1 servers had a hardware failure
that resulted in an unclean restart. When the server came back up 5 minutes
later, there were two topic corruption problems that we had to handle to
get the pipeline working again.

1. The kafka log directory had malformed partition directory names [1]. The
partition directories were shortened names of our topic naming scheme.
Inside some of these directories was a 0 byte .kafka file. This prevented
the kafka server process from starting. To resolve this, we manually
removed the malformed topic directories and the empty log files.

Is this a known issue? If so, has it been addressed in 0.8? This seems like
a bug.

2. One of the partitions had a corrupt log message (crc32 check failure)
that prevented consumers from advancing on that partition [2]. To resolve
this issue we had to hexdump the log file seeking to the byte offset
provided by the consumer exception and remove the corrupt message at the
frame boundary. After removing the corrupt message, we created a task to
parse the log file and put it back into our producer / consumer pipeline.
The InvalidMessageException never bubbled to our consumer thread,
preventing us from handling this error without manual manipulation of the
log files (our consumer code was catching InvalidMessageException, but it
never reached that point).

Is a corrupt log file a state that Kafka is supposed to handle, or is our
consumer code supposed to handle it? I'm confused about how we can avoid
manually cleaning up log files with a hex editor if we get into this state
again.

Thanks!

Blake

[1] Corrupt partition directories in the broker data directories:

0 [main] INFO kafka.utils.Utils$ - The number of partitions for
topic datapoints-write : 50
1 [main] INFO kafka.utils.Utils$ - The number of partitions for
topic datapoints-increment : 50
2 [main] INFO kafka.utils.Utils$ - The number of partitions for
topic series-index : 5
5 [main] INFO kafka.server.KafkaServer - Starting Kafka server...
17 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-18'
32 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-18/00000000095029068903.kafka in mutable
mode, recovery false
46 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-15'
46 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-15/00000000000000000000.kafka in
mutable mode, recovery false
47 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-25'
47 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-25/00000000000000000000.kafka in
mutable mode, recovery false
47 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-27'
48 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-27/00000000093417873484.kafka in mutable
mode, recovery false
48 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-9'
48 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-9/00000000092880448733.kafka in mutable
mode, recovery false
49 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-8'
49 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-8/00000000000000000000.kafka in
mutable mode, recovery false
49 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-29'
49 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-29/00000000008053064625.kafka in
mutable mode, recovery false
50 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-34'
50 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-34/00000000000000000000.kafka in
mutable mode, recovery false
50 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-19'
51 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-19/00000000000000000000.kafka in
mutable mode, recovery false
51 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-16'
51 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-16/00000000000000000000.kafka in
mutable mode, recovery false
51 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-27'
52 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-27/00000000000000000000.kafka in
mutable mode, recovery false
52 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-34'
52 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-34/00000000092880904403.kafka in mutable
mode, recovery false
53 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-21'
53 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-21/00000000095028969469.kafka in mutable
mode, recovery false
53 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-48'
54 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-48/00000000094491048876.kafka in mutable
mode, recovery false
54 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-40'
55 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-40/00000000000000000000.kafka in
mutable mode, recovery false
55 [main] INFO kafka.log.LogManager - Loading log 'datapoints-increment-6'
55 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-increment-6/00000000000000000000.kafka in
mutable mode, recovery false
56 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-25'
56 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-25/00000000093417394574.kafka in mutable
mode, recovery false
56 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-33'
57 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-33/00000000093417473922.kafka in mutable
mode, recovery false
57 [main] INFO kafka.log.LogManager - Loading log 'datapoints-write-35'
58 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-write-35/00000000092880891040.kafka in mutable
mode, recovery false
58 [main] INFO kafka.log.LogManager - Loading log 'datapoints-wr'
58 [main] INFO kafka.log.Log - Loading the last segment
/var/kafka/datapoints-wr/00000000000000000000.kafka in mutable mode,
recovery false
60 [main] FATAL kafka.server.KafkaServerStartable - Fatal error
during KafkaServerStable startup. Prepare to shutdown
java.lang.NumberFormatException: For input string: "wr"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:481)
at java.lang.Integer.parseInt(Integer.java:514)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:207)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
at kafka.utils.Utils$.getTopicPartition(Utils.scala:558)
at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:71)
at kafka.log.LogManager$$anonfun$4.apply(LogManager.scala:65)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:34)
at kafka.log.LogManager.<init>(LogManager.scala:65)
at kafka.server.KafkaServer.startup(KafkaServer.scala:58)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:50)
at kafka.Kafka.main(Kafka.scala)
61 [main] INFO kafka.server.KafkaServer - Shutting down Kafka server
62 [main] INFO kafka.utils.KafkaScheduler - shutdown scheduler
kafka-logcleaner-
62 [main] INFO kafka.server.KafkaServer - Kafka server shut down completed

[blake [ at ] kafka03 kafka]$ ls
dat datapoints-increment-40 datapoints-write-26
datapoints-increment-0 datapoints-increment-41 datapoints-write-27
datapoints-increment-1 datapoints-increment-42 datapoints-write-28
datapoints-increment-10 datapoints-increment-43 datapoints-write-29
datapoints-increment-11 datapoints-increment-44 datapoints-write-3
datapoints-increment-12 datapoints-increment-45 datapoints-write-30
datapoints-increment-13 datapoints-increment-46 datapoints-write-31
datapoints-increment-14 datapoints-increment-47 datapoints-write-32
datapoints-increment-15 datapoints-increment-48 datapoints-write-33
datapoints-increment-16 datapoints-increment-49 datapoints-write-34
datapoints-increment-17 datapoints-increment-5 datapoints-write-35
datapoints-increment-18 datapoints-increment-6 datapoints-write-36
datapoints-increment-19 datapoints-increment-7 datapoints-write-37
datapoints-increment-2 datapoints-increment-8 datapoints-write-38
datapoints-increment-20 datapoints-increment-9 datapoints-write-39
datapoints-increment-21 datapoints-wr datapoints-write-4
datapoints-increment-22 datapoints-writ datapoints-write-40
datapoints-increment-23 datapoints-write-0 datapoints-write-41
datapoints-increment-24 datapoints-write-1 datapoints-write-42
datapoints-increment-25 datapoints-write-10 datapoints-write-43
datapoints-increment-26 datapoints-write-11 datapoints-write-44
datapoints-increment-27 datapoints-write-12 datapoints-write-45
datapoints-increment-28 datapoints-write-13 datapoints-write-46
datapoints-increment-29 datapoints-write-14 datapoints-write-47
datapoints-increment-3 datapoints-write-15 datapoints-write-48
datapoints-increment-30 datapoints-write-16 datapoints-write-49
datapoints-increment-31 datapoints-write-17 datapoints-write-5
datapoints-increment-32 datapoints-write-18 datapoints-write-6
datapoints-increment-33 datapoints-write-19 datapoints-write-7
datapoints-increment-34 datapoints-write-2 datapoints-write-8
datapoints-increment-35 datapoints-write-20 datapoints-write-9
datapoints-increment-36 datapoints-write-21 series-index-0
datapoints-increment-37 datapoints-write-22 series-index-1
datapoints-increment-38 datapoints-write-23 series-index-2
datapoints-increment-39 datapoints-write-24 series-index-3
datapoints-increment-4 datapoints-write-25 series-index-4
[blake [ at ] kafka03 kafka]$ cd datapoints-wr
[blake [ at ] kafka03 datapoints-wr]$ ls -lah
total 8.0K
drwxr-xr-x 2 kafka kafka 4.0K Jul 11 18:51 .
drwxr-xr-x 110 kafka kafka 4.0K Jul 11 19:05 ..
-rw-r--r-- 1 kafka kafka 0 Jul 11 18:51 00000000000000000000.kafka

[2] In our consumer logs:

013-07-11 21:33:01,136 61867 [FetchRunnable-0] ERROR
kafka.consumer.FetcherRunnable - error in FetcherRunnable
kafka.message.InvalidMessageException: message is invalid, compression
codec: NoCompressionCodec size: 79 curr offset: 93706191248 init
offset: 93706191165
at kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:130)
at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:160)
at kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:100)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51)
at kafka.message.ByteBufferMessageSet.shallowValidBytes(ByteBufferMessageSet.scala:64)
at kafka.message.ByteBufferMessageSet.validBytes(ByteBufferMessageSet.scala:59)
at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:57)
at kafka.consumer.FetcherRunnable$$anonfun$run$6.apply(FetcherRunnable.scala:79)
at kafka.consumer.FetcherRunnable$$anonfun$run$6.apply(FetcherRunnable.scala:65)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:76)
at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:65)

Message Encoding in Kafka 0.8

July 16, 2013, 9:56 pm

≫ Next: Duplicate Messages on the Consumer

≪ Previous: Topic corruption from hardware failure (0.7.1)

$

0

0

I noticed that ProducerData has been removed in 0.8 branch

If I'd wish to send a complex message type (encoded message) ,

how would I do it in 0.8?

In 0.7.X (a snippet from Neha's example)

val producer = new Producer[Message, MemberRecord](config);

// send a single message
val message = new MemberRecord(1, "John", "US")
val producerData = new ProducerData[Message,
MemberRecord]("member-records", message)
producer.send(producerData)

class MemberRecord(val memberId: Int, val name: String, val location: String) {
override def toString = {
"(" + memberId + "," + name + "," + location + ")"

class MemberRecordEncoder extends Encoder[MemberRecord] {
def toMessage(member: MemberRecord):Message = {
val outputStream = new ByteArrayOutputStream()
val dos = new DataOutputStream(outputStream)
dos.writeInt(member.memberId)
dos.writeUTF(member.name)
dos.writeUTF(member.location)
outputStream.flush

new Message(outputStream.toByteArray)

Duplicate Messages on the Consumer

July 18, 2013, 8:55 am

≫ Next: Meetup in Raleigh/Durham, NC

≪ Previous: Message Encoding in Kafka 0.8

$

0

0

Hello,

We recently started seeing duplicate messages appearing at our consumers. Thankfully, the database is set up so that we don't store the dupes, but it is annoying. It's not every message, only about 1% of them. We are running 0.7.0 for the broker with Zookeeper 3.3.4 from Cloudera and 0.7.0 for the producer and consumer. We tried upgrading the consumer to 0.7.2 to see if that worked, but we're still seeing the dupes. Do we have to upgrade the broker as well to resolve this? Is there something we can check to see what's going on because we're not seeing anything unusual in the logs. I suspected that there may be significant rebalancing, but that does not appear to be the case at all.

Casey Sybrandy

Meetup in Raleigh/Durham, NC

July 18, 2013, 9:19 pm

≫ Next: Retrieve most-recent-n messages from kafka topic

≪ Previous: Duplicate Messages on the Consumer

$

0

0

There is a Hadoop meetup happening in Durham next week. I'm presenting
an intro to Kafka.

I don't suspect there are any Kafka users in the area who are not
already members of TriHUG, but sending this email out just in case :)

http://www.meetup.com/TriHUG/events/129831822/

-David

---

Retrieve most-recent-n messages from kafka topic

July 19, 2013, 12:45 am

≫ Next: Monitoring with Riemann

≪ Previous: Meetup in Raleigh/Durham, NC

$

0

0

We're running Kafka 0.7 and I'm hitting some issues trying to access the
newest n messages in a topic (or at least in a broker/partition combo) and
wondering if my use case just isn't supported or if I'm missing something.
What I'd like to be able to do is get the most recent offset from a
broker/partition combo, subtract an amount of bytes roughly equivalent to
messages_desired*bytes_per_message and then issue a FetchRequest with that
offset and amount of bytes.

I gathered from this
post<http://mail-archives.apache.org/mod_mbox/kafka-users/201212.mbox/%3CCCF8F23D.5E4A%25zhaoyonghui [ at ] gmail.com%3E>
that
I need to use the Simple Consumer in order to do offset manipulation beyond
the start from beginning and start from end options. And I saw from this
post<http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201209.mbox/%3CCALD69j0iDCZZFF3nm-wrfvW5Y6wwxRZFOL8A1QQFugQUKdo6Xg [ at ] mail.gmail.com%3E>
that
the offsets returned by getOffsetsBefore are really only the major
checkpoints when files are rolled over, every 500MB by default. I also
found that if I take an offset returned from getOffsetsBefore and subtract
a fixed value, say 100KB, and submit that offset with a FetchRequest I get
a kafka.common.InvalidMessageSizeException, presumably since my computed
offset didn't align with a real message offset.

As far as I can tell, this leaves me only able to find the most recent
milestone offset, perhaps up to 500MB behind current data, and extract a
batch from that point forward. Is there any other way that I'm missing
here? The two things that seem to be lacking are access to the most recent
offset and the ability to rollback from that offset by a fixed amount of
bytes or messages without triggering the InvalidMessageSizeException.

Thanks,
Shane

Monitoring with Riemann

July 18, 2013, 2:00 pm

≫ Next: Flexible Producers and Consumers

≪ Previous: Retrieve most-recent-n messages from kafka topic

$

0

0

Hi,

Earlier today I finished work on a library that adds an additional metrics
reporter (https://github.com/pingles/kafka-riemann-reporter). I did it
because although we were writing the CSV metrics we had nothing to display
the data.

It pushes metrics to Riemann (http://riemann.io) a very cool tool for
aggregating and processing event streams. We've experimented with it a
little in the past but this is the first production integration we've had.
Streams are processed by Riemann and then plotted by Librato (
http://librato.com).

Installing requires building the project and copying the JAR to Kafka's
./libs dir and then putting the following in the server.properties:

kafka.metrics.reporters=org.pingles.kafka.KafkaRiemannReporter
kafka.riemann.metrics.reporter.enabled=true
kafka.riemann.metrics.reporter.publisher=riemann
kafka.riemann.metrics.reporter.publisher.host=127.0.0.1
kafka.riemann.metrics.reporter.publisher.port=5555

It lets you push directly from the broker to a Riemann service. However, we
actually push messages (the Riemann events- protocol buffer messages) onto
a Kafka topic to be later consumed by a bridged and forwarded to Riemann.
This has the added benefit of allowing metrics to be flowed into our
existing Redshift data load pipeline as well as being monitored in realtime.

It's pretty early days but I deployed it to our small Kafka cluster on EC2
earlier and have been watching the graphs all day. Although we were writing
metrics to CSVs before we had nothing to visualise it, this makes it pretty
easy to pump the broker metrics out.

I thought I'd mention it here in case other people are in a similar
position to us and wanted an easy (alebit still a little experimental :)
way to get the metrics out of Kafka.

I'd really appreciate some feedback, and of course I'd be delighted if
other people use it and help fix my mistakes :)

At the moment it's built against an earlier 0.8 that we're running in
production (9bd2a) but I'm keen to start working (and deploying) an
officially released build.

Paul

Flexible Producers and Consumers

July 16, 2013, 11:31 am

≫ Next: Plz help me

≪ Previous: Monitoring with Riemann

$

0

0

I've read through many sites regarding using Kafka for log aggregation but haven't really found anything that actually talks about how people are shipping their logs into Kafka and consuming them. I'm really interested in implementation that would watch any kind of logs (local syslogs and application logs) and ship them near realtime into kafka. I think products like logstash and flume really shine in this area as they have multitude of options to ship any data stream into central aggregation service.

Since Kafka is proclaimed to be far more scalable I'm hoping there are options such as (http://logstash.net/docs/1.1.13/) to be able to vacuum any data source and put it into kafka queues and then consume them.

Any suggestions?
[cid:41F2EB5A-543E-42B3-B328-54EA3808E32D]

Plz help me

July 16, 2013, 12:06 am

≫ Next: Plz help me

≪ Previous: Flexible Producers and Consumers

$

0

0

I have 2 kafka and 1 zookeeper

But When I use kafka producer to send msg to zookeeper ,It just send msg to
broker 0,but not broker 1.

./kafka/bin/kafka-producer-perf-test.sh --brokerinfo zk.connect=
211.43.180.171:2181 --messages 100000 --topic my_topic --threads 2
--message-size 200 --async=true

*2013-07-16 15:59:55,158] INFO Creating async producer for broker id = 1 at
10.10.6.201:9092(kafka.producer.ProducerPool)
[2013-07-16 15:59:55,158] INFO Creating async producer for broker id = 0 at
10.10.7.166:9092(kafka.producer.ProducerPool)*
[2013-07-16 15:59:55,159] INFO Producer properties = {zk.connect=
211.43.180.171:2181, compression.codec=0, buffer.size=65536,
reconnect.interval=2147483647, batch.size=200, producer.type=async,
queue.enqueueTimeout.ms <http://queue.enqueuetimeout.ms/>=-1} (kafka.tools.
ProducerPerformance$AsyncProducerThread)
[2013-07-16 15:59:55,159] INFO Messages per thread = 50000 (kafka.tools.
ProducerPerformance$AsyncProducerThread)
[2013-07-16 15:59:55,159] INFO Initiating client connection, connectString=
211.43.180.171:2181sessionTimeout=6000 watcher=org.I0Itec.zkclient.
ZkClient [ at ] 17bc72c0 (org.apache.zookeeper.ZooKeeper)
[2013-07-16 15:59:55,160] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2013-07-16 15:59:55,167] INFO Opening socket connection to server /
211.43.180.171:2181(org.apache.zookeeper.ClientCnxn)
*[2013-07-16 15:59:55,417] INFO Connected to 10.10.7.166:9092 for producing
(kafka.producer.SyncProducer)*
[2013-07-16 15:59:56,656] INFO thread 0: 5000 messages sent 3351.2064
nMsg/sec 2.1521 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:57,323] INFO thread 0: 10000 messages sent 7440.4762
nMsg/sec 4.8249 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:57,774] INFO thread 0: 15000 messages sent 11086.4745
nMsg/sec 7.1401 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:57,970] INFO thread 0: 20000 messages sent 25510.2041
nMsg/sec 16.3751 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:58,101] INFO thread 0: 25000 messages sent 38167.9389
nMsg/sec 24.7269 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:58,250] INFO thread 0: 30000 messages sent 33557.0470
nMsg/sec 21.6650 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:58,395] INFO thread 0: 35000 messages sent 34722.2222
nMsg/sec 22.4402 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:58,530] INFO thread 0: 40000 messages sent 36764.7059
nMsg/sec 23.6089 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:58,687] INFO thread 0: 45000 messages sent 31847.1338
nMsg/sec 20.4403 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:58,797] INFO thread 0: 50000 messages sent 45454.5455
nMsg/sec 29.1881 MBs/sec (kafka.tools.ProducerPerformance$
AsyncProducerThread)
[2013-07-16 15:59:58,797] INFO Closing all async producers
(kafka.producer.ProducerPool)

producers always connect to broker 0,I don't know why

Plz help me.

Plz help me

July 16, 2013, 12:04 am

≫ Next: Replacing brokers in a cluster (0.8)

≪ Previous: Plz help me

$

0

0

I have 2 kafka and 1 zookeeper

But When I use kafka producer to send msg to zookeeper ,It just send msg to
broker 0,but not broker 1.

./kafka/bin/kafka-producer-perf-test.sh --brokerinfo zk.connect=
211.43.180.171:2181 --messages 100000 --topic my_topic --threads 2
--message-size 200 --async=true

Replacing brokers in a cluster (0.8)

July 22, 2013, 2:14 am

≫ Next: Apache Kafka Question

≪ Previous: Plz help me

$

0

0

I'm planning to upgrade a 0.8 cluster from 2 old nodes, to 3 new ones
(better hardware). I'm using a replication factor of 2.

I'm thinking the plan should be to spin up the 3 new nodes, and operate as
a 5 node cluster for a while. Then first remove 1 of the old nodes, and
wait for the partitions on the removed node to get replicated to the other
nodes. Then, do the same for the other old node.

Does this sound sensible?

How does the cluster decide when to re-replicate partitions that are on a
node that is no longer available? Does it only happen if/when new messages
arrive for that partition? Is it on a partition by partition basis?

Or is it a cluster-level decision that a broker is no longer valid, in
which case all affected partitions would immediately get replicated to new
brokers as needed?

I'm just wondering how I will know when it will be safe to take down my
second old node, after the first one is removed, etc.

Thanks,

Jason

Apache Kafka Question

July 22, 2013, 3:57 am

≫ Next: Logo

≪ Previous: Replacing brokers in a cluster (0.8)

$

0

0

Hi,

I am planning to use Apache Kafka 0.8 to handle millions of messages per day. Now I need to form the environment, like

(i) How many Topics to be created?
(ii) How many partitions/replications to be created?
(iii) How many Brokers to be created?
(iv) How many consumer instances in consumer group?

(v) Topic or Queue? If topic whether we need to create multiple group Id as supposed to single one?

How we can go about it? Please clarify.

Thanks & Regards,
Anantha

Please do not print this email unless it is absolutely necessary.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

www.wipro.com

---

Logo

July 22, 2013, 9:51 am

≫ Next: Messages TTL setting

≪ Previous: Apache Kafka Question

$

0

0

Hey guys,

We need a logo!

I got a few designs from a 99 designs contest that I would like to put
forward:
https://issues.apache.org/jira/browse/KAFKA-982

If anyone else would like to submit a design that would be great.

Let's do a vote to choose one.

-Jay

Messages TTL setting

July 22, 2013, 2:03 pm

≫ Next: Recommended log level in prod environment.

≪ Previous: Logo

$

0

0

Hi,

We have a 3 node Kafka cluster. We want to increase the maximum amount of
time for which messages are saved in Kafka data logs.
Can we change the configuration on one node, stop it and start it and then
change the configuration of the next node?
Or should we stop all 3 nodes at a time, make configuration changes and
then restart all 3? Please suggest.

Thanks,
Arathi

Recommended log level in prod environment.

July 22, 2013, 6:43 pm

≫ Next: clean up kafka environmet

≪ Previous: Messages TTL setting

$

0

0

The beta release comes with mostly trace level logging. Is this
recommended? I notice our cluster produce way too many logs. I set all the
level to info currently.

clean up kafka environmet

July 23, 2013, 7:46 am

≫ Next: Consumer stops consuming after 5 gb

≪ Previous: Recommended log level in prod environment.

$

0

0

Hi All.

I have on one machine kafka installation. I needed to move it to another
machine and I copied a kafka folder to that machine.

when I started kafka in new machine I got such output:

[2013-07-23 17:03:29,858] INFO Got user-level KeeperException when
processing sessionid:0x1400bd6e2960000 type:create cxid:0x1
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/ids
Error:KeeperErrorCode = NoNode for /brokers/ids
(org.apache.zookeeper.server.PrepRequestProcessor)
[2013-07-23 17:03:29,872] INFO Got user-level KeeperException when
processing sessionid:0x1400bd6e2960000 type:create cxid:0x2
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers
Error:KeeperErrorCode = NoNode for /brokers
(org.apache.zookeeper.server.PrepRequestProcessor)
[2013-07-23 17:03:29,886] INFO Registering broker /brokers/ids/0 succeeded
with id:0,creatorId:172.31.202.2-1374588209838,host:172.31.202.2,port:9092
(kafka.server.KafkaZooKeeper)
[2013-07-23 17:03:29,907] INFO Starting log flusher every 1000 ms with the
following overrides Map() (kafka.log.LogManager)
[2013-07-23 17:03:29,909] INFO Kafka server started.
(kafka.server.KafkaServer)
[2013-07-23 17:15:35,903] INFO Accepted socket connection from
/0:0:0:0:0:0:0:1:45710 (org.apache.zookeeper.server.NIOServerCnxn)
[2013-07-23 17:15:35,906] INFO Client attempting to establish new session
at /0:0:0:0:0:0:0:1:45710 (org.apache.zookeeper.server.NIOServerCnxn)
[2013-07-23 17:15:35,909] INFO Established session 0x1400bd6e2960001 with
negotiated timeout 6000 for client /0:0:0:0:0:0:0:1:45710
(org.apache.zookeeper.server.NIOServerCnxn)
[2013-07-23 17:35:19,019] INFO Created log for 'topic1'-0
(kafka.log.LogManager)
[2013-07-23 17:35:19,023] INFO Begin registering broker topic
/brokers/topics/topic1/0 with 1 partitions (kafka.server.KafkaZooKeeper)
[2013-07-23 17:35:19,024] INFO Got user-level KeeperException when
processing sessionid:0x1400bd6e2960000 type:create cxid:0x6
zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error
Path:/brokers/topics/topic1 Error:KeeperErrorCode = NoNode for
/brokers/topics/topic1 (org.apache.zookeeper.server.PrepRequestProcessor)
[2013-07-23 17:35:19,032] INFO End registering broker topic
/brokers/topics/topic1/0 (kafka.server.KafkaZooKeeper)

It actually make sense getting exception related to the "Path:/brokers/ids".

Questions:
What should I do to fix the problem on my new machine? Is there and way
to cleanup kafka on new machine ?

Thanks
Oleg.

Consumer stops consuming after 5 gb

July 23, 2013, 7:58 am

≫ Next: found that the producer called localhost: 9092 in Kafka of Java client.

≪ Previous: clean up kafka environmet

$

0

0

After consuming about 5 gb of messages , it stopped consuming and it got
stuck at a particular offset. After running the following command I got an
error regarding the broker.

$ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181--topic air

[2013-07-23 20:19:42,748] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2013-07-23 20:19:42,757] INFO Client
environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,757] INFO Client environment:host.name=localhost
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,757] INFO Client environment:java.version=1.7.0_21
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,757] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,757] INFO Client
environment:java.home=/usr/java/jdk1.7.0_21/jre
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,757] INFO Client
environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client
environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client
environment:os.version=2.6.32-279.22.1.el6.x86_64
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client environment:user.name=cleartrip
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client environment:user.home=/home/cleartrip
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,758] INFO Client
environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,759] INFO Initiating client connection, connectString=
172.16.18.65:2181 sessionTimeout=30000
watcher=org.I0Itec.zkclient.ZkClient [ at ] 6926e27(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:19:42,783] INFO Opening socket connection to server /
172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn)
[2013-07-23 20:19:42,788] INFO Socket connection established to zoo1/
172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2013-07-23 20:19:42,800] INFO Session establishment complete on server
zoo1/172.16.18.65:2181, sessionid = 0x13fc7235423015f, negotiated timeout =
30000 (org.apache.zookeeper.ClientCnxn)
[2013-07-23 20:19:42,802] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
air-DummyProductionConsumerGroup-After,air,1-0
(Group,Topic,BrokerId-PartitionId)
Owner =
air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0
Consumer offset = 1140946916
= 1,140,946,916 (1.06G)
[2013-07-23 20:19:42,948] ERROR Could not parse broker info
d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092
(kafka.tools.ConsumerOffsetChecker$)
air-DummyProductionConsumerGroup-After,air,1-1
(Group,Topic,BrokerId-PartitionId)
Owner =
air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1
Consumer offset = 1142891729
= 1,142,891,729 (1.06G)
air-DummyProductionConsumerGroup-After,air,2-0
(Group,Topic,BrokerId-PartitionId)
Owner =
air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0
Consumer offset = 1587266466
= 1,587,266,466 (1.48G)
[2013-07-23 20:19:42,956] ERROR Could not parse broker info
d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092
(kafka.tools.ConsumerOffsetChecker$)
air-DummyProductionConsumerGroup-After,air,2-1
(Group,Topic,BrokerId-PartitionId)
Owner =
air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1
Consumer offset = 1566027800
= 1,566,027,800 (1.46G)
BROKER INFO
[2013-07-23 20:19:42,988] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2013-07-23 20:19:42,992] INFO EventThread shut down
(org.apache.zookeeper.ClientCnxn)
[2013-07-23 20:19:42,992] INFO Session: 0x13fc7235423015f closed
(org.apache.zookeeper.ZooKeeper)
[cleartrip [ at ] d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$ clear

[cleartrip [ at ] d1c2-analytics-r13-4 kafka-0.7.2-incubating-src]$
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
air-DummyProductionConsumerGroup-After --zkconnect 172.16.18.65:2181--topic air
[2013-07-23 20:20:12,669] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2013-07-23 20:20:12,679] INFO Client
environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,679] INFO Client environment:host.name=localhost
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,679] INFO Client environment:java.version=1.7.0_21
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,679] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,679] INFO Client
environment:java.home=/usr/java/jdk1.7.0_21/jre
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,679] INFO Client
environment:java.class.path=:bin/../project/boot/scala-2.8.0/lib/scala-compiler.jar:bin/../project/boot/scala-2.8.0/lib/scala-library.jar:bin/../core/target/scala_2.8.0/kafka-0.7.2.jar:bin/../core/lib/*.jar:bin/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:bin/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:bin/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:bin/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,679] INFO Client
environment:java.library.path=/usr/local/oracle:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,680] INFO Client environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,680] INFO Client environment:java.compiler=<NA>
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,680] INFO Client environment:os.name=Linux
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,680] INFO Client environment:os.arch=amd64
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,680] INFO Client
environment:os.version=2.6.32-279.22.1.el6.x86_64
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,680] INFO Client environment:user.name=cleartrip
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,680] INFO Client environment:user.home=/home/cleartrip
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,680] INFO Client
environment:user.dir=/home/cleartrip/kafka/kafka-0.7.2-incubating-src
(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,681] INFO Initiating client connection, connectString=
172.16.18.65:2181 sessionTimeout=30000
watcher=org.I0Itec.zkclient.ZkClient [ at ] 6926e27(org.apache.zookeeper.ZooKeeper)
[2013-07-23 20:20:12,705] INFO Opening socket connection to server /
172.16.18.65:2181 (org.apache.zookeeper.ClientCnxn)
[2013-07-23 20:20:12,710] INFO Socket connection established to zoo1/
172.16.18.65:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2013-07-23 20:20:12,719] INFO Session establishment complete on server
zoo1/172.16.18.65:2181, sessionid = 0x13fc72354230160, negotiated timeout =
30000 (org.apache.zookeeper.ClientCnxn)
[2013-07-23 20:20:12,722] INFO zookeeper state changed (SyncConnected)
(org.I0Itec.zkclient.ZkClient)
air-DummyProductionConsumerGroup-After,air,1-0
(Group,Topic,BrokerId-PartitionId)
Owner =
air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-0
Consumer offset = 1140946916
= 1,140,946,916 (1.06G)
[2013-07-23 20:20:12,828] ERROR Could not parse broker info
d1c2-analytics-r13-4-1374557209186:d1c2-analytics-r13-4:9092
(kafka.tools.ConsumerOffsetChecker$)
air-DummyProductionConsumerGroup-After,air,1-1
(Group,Topic,BrokerId-PartitionId)
Owner =
air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589085082-f88ba95e-1
Consumer offset = 1142891729
= 1,142,891,729 (1.06G)
air-DummyProductionConsumerGroup-After,air,2-0
(Group,Topic,BrokerId-PartitionId)
Owner =
air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-0
Consumer offset = 1587266466
= 1,587,266,466 (1.48G)
[2013-07-23 20:20:12,835] ERROR Could not parse broker info
d1c2-analytics-r13-5-1374557226228:d1c2-analytics-r13-5:9092
(kafka.tools.ConsumerOffsetChecker$)
air-DummyProductionConsumerGroup-After,air,2-1
(Group,Topic,BrokerId-PartitionId)
Owner =
air-DummyProductionConsumerGroup-After_d1c2-analytics-r13-17.cltp.com-1374589174762-cd80342e-1
Consumer offset = 1566027800
= 1,566,027,800 (1.46G)
BROKER INFO
[2013-07-23 20:20:12,867] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2013-07-23 20:20:12,871] INFO EventThread shut down
(org.apache.zookeeper.ClientCnxn)
[2013-07-23 20:20:12,871] INFO Session: 0x13fc72354230160 closed
(org.apache.zookeeper.ZooKeeper)