Hi, I am new with kafka and using kafka 0.8 to build a distributed queuing
system in amazon web service cluster.
I have 4 machines Z1, B1, B2 and B3. 1 Zookeeper instance is running on Z1
and 3 different brokers are running on B1,B2 and B3 respectively.
I am running 3 producers on 3 broker machines(B1, B2, B3) , one in each
machine. Similarly 3 consumers on 3 broker machines, one in each machine.
I created a topic , lets say 'test', with 12 partitions (test-0,test-1 ...
test-11).
4 partitions in each broker machine.
B1 - test-0,test-1,test-2,test-3
B2 - test-4,test-5,test-6,test-7
B3 - test-8,test-9,test-10,test-11
Zookeeper assigned broker in each machine as a leader to the partitions
present in the same machine.
Partition - leader
test-0 - B1
test-1 - B1
test-2 - B1
test-3 - B1
test-4 - B2
test-5 - B2
test-6 - B2
test-7 - B2
test-8 - B3
test-9 - B3
test-10 - B3
test-11 - B3
All 3 producers are producing messages to this topic 'test' and all 3
consumers are trying to consume from the same topic 'test'.
What I am trying to achieve here is , whenever a producer send a message to
this topic , it should use the broker present in the same machine as
producer and ultimately using the partitions in the same machine.
Producer 1 ---> B1 ----> (test-0,test-1,test-2,test-3) -----> consumer 1
Producer 2 ---> B2 ----> (test-4,test-5,test-6,test-7) -----> consumer 2
Producer 3 ---> B3 ----> (test-8,test-9,test-10,test-11) -----> consumer 3
I am assuming this will reduce the inter-machine message transfer and will
improve the performance.
My questions are :
1) Does it really help in improving performance, when message is produced
and consumed from same machine in a distributed environment.
2) I read that producer can fetch metadata from broker about all
leader-partition mapping for a topic. It will help to pick the leader
present in the same machine as producer. How a producer can fetch this
metadata ? Could not find any implementation.
Thanks in advance,
Abhijeet
system in amazon web service cluster.
I have 4 machines Z1, B1, B2 and B3. 1 Zookeeper instance is running on Z1
and 3 different brokers are running on B1,B2 and B3 respectively.
I am running 3 producers on 3 broker machines(B1, B2, B3) , one in each
machine. Similarly 3 consumers on 3 broker machines, one in each machine.
I created a topic , lets say 'test', with 12 partitions (test-0,test-1 ...
test-11).
4 partitions in each broker machine.
B1 - test-0,test-1,test-2,test-3
B2 - test-4,test-5,test-6,test-7
B3 - test-8,test-9,test-10,test-11
Zookeeper assigned broker in each machine as a leader to the partitions
present in the same machine.
Partition - leader
test-0 - B1
test-1 - B1
test-2 - B1
test-3 - B1
test-4 - B2
test-5 - B2
test-6 - B2
test-7 - B2
test-8 - B3
test-9 - B3
test-10 - B3
test-11 - B3
All 3 producers are producing messages to this topic 'test' and all 3
consumers are trying to consume from the same topic 'test'.
What I am trying to achieve here is , whenever a producer send a message to
this topic , it should use the broker present in the same machine as
producer and ultimately using the partitions in the same machine.
Producer 1 ---> B1 ----> (test-0,test-1,test-2,test-3) -----> consumer 1
Producer 2 ---> B2 ----> (test-4,test-5,test-6,test-7) -----> consumer 2
Producer 3 ---> B3 ----> (test-8,test-9,test-10,test-11) -----> consumer 3
I am assuming this will reduce the inter-machine message transfer and will
improve the performance.
My questions are :
1) Does it really help in improving performance, when message is produced
and consumed from same machine in a distributed environment.
2) I read that producer can fetch metadata from broker about all
leader-partition mapping for a topic. It will help to pick the leader
present in the same machine as producer. How a producer can fetch this
metadata ? Could not find any implementation.
Thanks in advance,
Abhijeet