Hi,
I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like
to add a new broker in the cluster. Each kafka instance has 400Go of data
and we are using replication factor equals to 3 with 50 partitions for each
topics.
I have check the documentation and especially the section dealing with the
cluster expansion (
https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion).
So far, I have tried the automatic way to reassign partitions using
the --topics-to-move-json-file
option on 1 small topic. Here is the command that I have ran:
content of topic-to-move.json:
"topics": [
{"topic": "SMALL_TOPIC"}],
"version":1
command:
./bin/kafka-reassign-partitions.sh --zookeeper zkDns:2181
--topics-to-move-json-file topics-to-move.json --broker-list
"1,2,3,4,5,6,7,8,9" --generate
Has you can see I specified a broker list including all the broker from the
cluster (including the newly added kafka node number 9).
First of all is it the right way to do that? As I am using replication
factor = 3, I have to specify at least 3 brokers. I thought that specifying
the entire list of broker will keep the cluster balanced (and it did).
Here is my main question, I looked the Ganglia ui in order to see the
impact of this command and here is the networking graph (
http://s18.postimg.org/hokyoeyax/kafka_network.png)
As you can see on the screen-shot the network traffic goes pretty high.
That is why I am afraid to run this tool on bigger topics.
I also took a look into the manual way of reassigning partitions. the
problem is that I always have to specify 3 replicas even if I just want to
move only one to my new broker.
Do you know any tool or script to help me in this process?
Thanks for your help,
Florian
I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like
to add a new broker in the cluster. Each kafka instance has 400Go of data
and we are using replication factor equals to 3 with 50 partitions for each
topics.
I have check the documentation and especially the section dealing with the
cluster expansion (
https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion).
So far, I have tried the automatic way to reassign partitions using
the --topics-to-move-json-file
option on 1 small topic. Here is the command that I have ran:
content of topic-to-move.json:
"topics": [
{"topic": "SMALL_TOPIC"}],
"version":1
command:
./bin/kafka-reassign-partitions.sh --zookeeper zkDns:2181
--topics-to-move-json-file topics-to-move.json --broker-list
"1,2,3,4,5,6,7,8,9" --generate
Has you can see I specified a broker list including all the broker from the
cluster (including the newly added kafka node number 9).
First of all is it the right way to do that? As I am using replication
factor = 3, I have to specify at least 3 brokers. I thought that specifying
the entire list of broker will keep the cluster balanced (and it did).
Here is my main question, I looked the Ganglia ui in order to see the
impact of this command and here is the networking graph (
http://s18.postimg.org/hokyoeyax/kafka_network.png)
As you can see on the screen-shot the network traffic goes pretty high.
That is why I am afraid to run this tool on bigger topics.
I also took a look into the manual way of reassigning partitions. the
problem is that I always have to specify 3 replicas even if I just want to
move only one to my new broker.
Do you know any tool or script to help me in this process?
Thanks for your help,
Florian