Hi, has anyone upgraded their kafka from 0.8.0 to 0.8.1 successfully one
broker at a time on a live cluster?
I am seeing strange behaviors where many of my kafka topics become unusable
(by both consumers and producers). When that happens, I see lots of errors
in the server logs that look like this:
[2014-04-09 10:38:14,669] WARN [KafkaApi-1007] Fetch request with
correlation id 2455 from client ReplicaFetcherThread-15-1007 on partition
[risk,0] failed due to Topic risk either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
[2014-04-09 10:38:14,669] WARN [KafkaApi-1007] Fetch request with
correlation id 2455 from client ReplicaFetcherThread-7-1007 on partition
[message,0] failed due to Topic message either doesn't exist or is in the
process of being deleted (kafka.server.KafkaApis)
When I try to consume a message from a topic that complained about the
Topic not existing (above warning), I get the below exception:
....topic message --from-beginning
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
[2014-04-09 10:40:30,571] WARN
[console-consumer-90716_dkafkadatahub07.tag-dev.com-1397065229615-7211ba72-leader-finder-thread],
Failed to add leader for partitions [message,0]; will retry
(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.UnknownTopicOrPartitionException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:79)
at
kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:167)
at
kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
at
kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
at
kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:119)
at
kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
at
kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
at
kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:119)
at
kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
at
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
*More details about my issues:*
My current configuration in the environment where I am testing the upgrade
is 4 physical servers running 2 brokers each with controlled shutdown
feature enabled. When I shutdown the 2 brokers on one of the existing Kafka
0.8.0 machines and upgrade that machine to 0.8.1 and restart it, all is
fine for a bit. Once, the new brokers come up, I ran the
kafka-preferred-replica-election.sh to make sure that started brokers
become leaders of existing topics. The replication factor on the topics is
set to 4. I tested both producing and consuming messages against brokers
that were leaders with kafka 0.8.0 and 0.8.1 and no issues were encountered.
Later, I tried to perform the control shutdown of the 2 additional brokers
on the Kafka server that has 0.8.0 version installed and after the broker
shutdown and new leaders were assigned, all of my server logs are getting
filled up with the above exceptions and most of my topics are not usable. I
have pulled and build the 0.8.1 kafka code from git last thursday so I
should be pretty much up to date. So not sure if I am doing something wrong
or if migrating from 0.8.0 to 0.8.1 on a live cluster one server at a time
is not supported. Is there a recommended migration approach that one should
take when migrating from live 0.8.0 to 0.8.1 cluster?
As to who is the leader of one of the topics that became unusable is the
broker that was successfully upgraded to 0.8.1:
Topic:message PartitionCount:1 ReplicationFactor:4 Configs:
Topic: message Partition: 0 * Leader: 1007 * Replicas:
1007,8,9,1001 Isr: 1001,1007,8
Brokers 9 and 1009 where shutdown from one physical server that had kafka
0.8.0 installed when these problems started occurring (I was planning to
upgrade them to 0.8.1). The only way I can recover from this state is to
shutdown all brokers and delete all of kafka topic logs plus zookeeper
kafka directory and start with new cluster.
Your help in this matter is greatly appreciated.
Thanks,
Martin
broker at a time on a live cluster?
I am seeing strange behaviors where many of my kafka topics become unusable
(by both consumers and producers). When that happens, I see lots of errors
in the server logs that look like this:
[2014-04-09 10:38:14,669] WARN [KafkaApi-1007] Fetch request with
correlation id 2455 from client ReplicaFetcherThread-15-1007 on partition
[risk,0] failed due to Topic risk either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
[2014-04-09 10:38:14,669] WARN [KafkaApi-1007] Fetch request with
correlation id 2455 from client ReplicaFetcherThread-7-1007 on partition
[message,0] failed due to Topic message either doesn't exist or is in the
process of being deleted (kafka.server.KafkaApis)
When I try to consume a message from a topic that complained about the
Topic not existing (above warning), I get the below exception:
....topic message --from-beginning
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
[2014-04-09 10:40:30,571] WARN
[console-consumer-90716_dkafkadatahub07.tag-dev.com-1397065229615-7211ba72-leader-finder-thread],
Failed to add leader for partitions [message,0]; will retry
(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.UnknownTopicOrPartitionException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:79)
at
kafka.consumer.SimpleConsumer.earliestOrLatestOffset(SimpleConsumer.scala:167)
at
kafka.consumer.ConsumerFetcherThread.handleOffsetOutOfRange(ConsumerFetcherThread.scala:60)
at
kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:179)
at
kafka.server.AbstractFetcherThread$$anonfun$addPartitions$2.apply(AbstractFetcherThread.scala:174)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:119)
at
kafka.server.AbstractFetcherThread.addPartitions(AbstractFetcherThread.scala:174)
at
kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:86)
at
kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:76)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:119)
at
kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:76)
at
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
*More details about my issues:*
My current configuration in the environment where I am testing the upgrade
is 4 physical servers running 2 brokers each with controlled shutdown
feature enabled. When I shutdown the 2 brokers on one of the existing Kafka
0.8.0 machines and upgrade that machine to 0.8.1 and restart it, all is
fine for a bit. Once, the new brokers come up, I ran the
kafka-preferred-replica-election.sh to make sure that started brokers
become leaders of existing topics. The replication factor on the topics is
set to 4. I tested both producing and consuming messages against brokers
that were leaders with kafka 0.8.0 and 0.8.1 and no issues were encountered.
Later, I tried to perform the control shutdown of the 2 additional brokers
on the Kafka server that has 0.8.0 version installed and after the broker
shutdown and new leaders were assigned, all of my server logs are getting
filled up with the above exceptions and most of my topics are not usable. I
have pulled and build the 0.8.1 kafka code from git last thursday so I
should be pretty much up to date. So not sure if I am doing something wrong
or if migrating from 0.8.0 to 0.8.1 on a live cluster one server at a time
is not supported. Is there a recommended migration approach that one should
take when migrating from live 0.8.0 to 0.8.1 cluster?
As to who is the leader of one of the topics that became unusable is the
broker that was successfully upgraded to 0.8.1:
Topic:message PartitionCount:1 ReplicationFactor:4 Configs:
Topic: message Partition: 0 * Leader: 1007 * Replicas:
1007,8,9,1001 Isr: 1001,1007,8
Brokers 9 and 1009 where shutdown from one physical server that had kafka
0.8.0 installed when these problems started occurring (I was planning to
upgrade them to 0.8.1). The only way I can recover from this state is to
shutdown all brokers and delete all of kafka topic logs plus zookeeper
kafka directory and start with new cluster.
Your help in this matter is greatly appreciated.
Thanks,
Martin