Quantcast
Channel: Kafka Timeline
Viewing all articles
Browse latest Browse all 1519

Kafka partition reassignment script

$
0
0
Hi,

I am currently running a Kafka 0.8.1.1 cluster with 8 servers. I would like
to add a new broker in the cluster. Each kafka instance has 400Go of data
and we are using replication factor equals to 3 with 50 partitions for each
topics.

I have check the documentation and especially the section dealing with the
cluster expansion (
https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion).

So far, I have tried the automatic way to reassign partitions using
the --topics-to-move-json-file
option on 1 small topic. Here is the command that I have ran:

content of topic-to-move.json:

"topics": [
{"topic": "SMALL_TOPIC"}],
"version":1

command:

./bin/kafka-reassign-partitions.sh --zookeeper zkDns:2181
--topics-to-move-json-file topics-to-move.json --broker-list
"1,2,3,4,5,6,7,8,9" --generate

Has you can see I specified a broker list including all the broker from the
cluster (including the newly added kafka node number 9).

First of all is it the right way to do that? As I am using replication
factor = 3, I have to specify at least 3 brokers. I thought that specifying
the entire list of broker will keep the cluster balanced (and it did).

Here is my main question, I looked the Ganglia ui in order to see the
impact of this command and here is the networking graph (
http://s18.postimg.org/hokyoeyax/kafka_network.png)

As you can see on the screen-shot the network traffic goes pretty high.
That is why I am afraid to run this tool on bigger topics.

I also took a look into the manual way of reassigning partitions. the
problem is that I always have to specify 3 replicas even if I just want to
move only one to my new broker.

Do you know any tool or script to help me in this process?

Thanks for your help,

Florian

Viewing all articles
Browse latest Browse all 1519

Trending Articles