Hi guys and jun,
We have a problem when adding a breakdown broker back to the cluster. Hope
you guys have some solution for it.
A cluster of 5 brokers(id=0~4) of kafka 0.8.0 was running for log
aggregation . Because of some issues of the disk, a broker(id=1) was down.
We spent one week to replace the disk, so we don't have any old data. When
adding the brokers back and using kafka-reassign-partitions.sh tool to
receive partition data from the up-to-date leaders, we have so many
warnings in server.log as shown below and no partition data files generated
in log.dirs.
[2014-01-15 20:00:00,519] WARN [Replica Manager on Broker 1]: While
recording the follower position, the partition [co,4] hasn't been created,
skip updating leader HW (kafka.server.ReplicaManager)
[2014-01-15 20:00:00,519] WARN [Replica Manager on Broker 1]: While
recording the follower position, the partition [co,9] hasn't been created,
skip updating leader HW (kafka.server.ReplicaManager)
[2014-01-15 20:00:00,519] WARN [KafkaApi-1] Fetch request with correlation
id 359929003 from client ReplicaFetcherThread-0-1 on partition [co,4]
failed due to Partition [co,4] doesn't exist on 1 (kafka.server.KafkaApis)
[2014-01-15 20:00:00,519] WARN [KafkaApi-1] Fetch request with correlation
id 359929003 from client ReplicaFetcherThread-0-1 on partition [co,9]
failed due to Partition [co,9] doesn't exist on 1 (kafka.server.KafkaApis)
[2014-01-15
And it seems an another issue in the listing topics result. Previously we
had configured 3 replica for the topic, but now there are only 2 replicas
for all the partitions and alive brokers.
topic: countinfo partition: 0 leader: 4 replicas: 3,4 isr: 4,3
topic: countinfo partition: 1 leader: 0 replicas: 4,0 isr: 0,4
topic: countinfo partition: 2 leader: 0 replicas: 0,2 isr: 0,2
topic: countinfo partition: 3 leader: 2 replicas: 2,3 isr: 2,3
topic: countinfo partition: 4 leader: 3 replicas: 3,0 isr: 3,0
topic: countinfo partition: 5 leader: 4 replicas: 4,2 isr: 4,2
topic: countinfo partition: 6 leader: 0 replicas: 0,3 isr: 0,3
topic: countinfo partition: 7 leader: 4 replicas: 2,4 isr: 4,2
topic: countinfo partition: 8 leader: 2 replicas: 3,2 isr: 2,3
topic: countinfo partition: 9 leader: 3 replicas: 4,3 isr: 3,4
So my question is: how can I re-add the breakdown broker back and make the
replicas to be 3 again ?
Thanks in advance.
Bo
We have a problem when adding a breakdown broker back to the cluster. Hope
you guys have some solution for it.
A cluster of 5 brokers(id=0~4) of kafka 0.8.0 was running for log
aggregation . Because of some issues of the disk, a broker(id=1) was down.
We spent one week to replace the disk, so we don't have any old data. When
adding the brokers back and using kafka-reassign-partitions.sh tool to
receive partition data from the up-to-date leaders, we have so many
warnings in server.log as shown below and no partition data files generated
in log.dirs.
[2014-01-15 20:00:00,519] WARN [Replica Manager on Broker 1]: While
recording the follower position, the partition [co,4] hasn't been created,
skip updating leader HW (kafka.server.ReplicaManager)
[2014-01-15 20:00:00,519] WARN [Replica Manager on Broker 1]: While
recording the follower position, the partition [co,9] hasn't been created,
skip updating leader HW (kafka.server.ReplicaManager)
[2014-01-15 20:00:00,519] WARN [KafkaApi-1] Fetch request with correlation
id 359929003 from client ReplicaFetcherThread-0-1 on partition [co,4]
failed due to Partition [co,4] doesn't exist on 1 (kafka.server.KafkaApis)
[2014-01-15 20:00:00,519] WARN [KafkaApi-1] Fetch request with correlation
id 359929003 from client ReplicaFetcherThread-0-1 on partition [co,9]
failed due to Partition [co,9] doesn't exist on 1 (kafka.server.KafkaApis)
[2014-01-15
And it seems an another issue in the listing topics result. Previously we
had configured 3 replica for the topic, but now there are only 2 replicas
for all the partitions and alive brokers.
topic: countinfo partition: 0 leader: 4 replicas: 3,4 isr: 4,3
topic: countinfo partition: 1 leader: 0 replicas: 4,0 isr: 0,4
topic: countinfo partition: 2 leader: 0 replicas: 0,2 isr: 0,2
topic: countinfo partition: 3 leader: 2 replicas: 2,3 isr: 2,3
topic: countinfo partition: 4 leader: 3 replicas: 3,0 isr: 3,0
topic: countinfo partition: 5 leader: 4 replicas: 4,2 isr: 4,2
topic: countinfo partition: 6 leader: 0 replicas: 0,3 isr: 0,3
topic: countinfo partition: 7 leader: 4 replicas: 2,4 isr: 4,2
topic: countinfo partition: 8 leader: 2 replicas: 3,2 isr: 2,3
topic: countinfo partition: 9 leader: 3 replicas: 4,3 isr: 3,4
So my question is: how can I re-add the breakdown broker back and make the
replicas to be 3 again ?
Thanks in advance.
Bo