Hi,
We got the following error spamming the logs when broker 1 is the controller and we are shutting it down in controlled manner not kill -9.
The leader being switched to broker 2 for all partitions but controller is not being failed over to broker 2.
[2014-04-16 10:48:47.976-0600] ERROR [Controller-1-to-broker-1-send-thread], Controller 1's connection to broker id:1,host:tm1-kafkabroker101,port:9092 was unsuccessful (kafka.controller.RequestSendThread)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-04-16 10:48:48.276-0600] ERROR [Controller-1-to-broker-1-send-thread], Controller 1 epoch 38 failed to send LeaderAndIsr request with correlation id 1766 to broker id:1,host:tm1-kafkabroker101,port:9092. Reconnecting to broker. (kafka.controller.RequestSendThread)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-04-16 10:48:48.277-0600] ERROR [Controller-1-to-broker-1-send-thread], Controller 1's connection to broker id:1,host:tm1-kafkabroker101,port:9092 was unsuccessful (kafka.controller.RequestSendThread)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
It stopped our upgrade because of this issue.
Thanks,
Balaji
We got the following error spamming the logs when broker 1 is the controller and we are shutting it down in controlled manner not kill -9.
The leader being switched to broker 2 for all partitions but controller is not being failed over to broker 2.
[2014-04-16 10:48:47.976-0600] ERROR [Controller-1-to-broker-1-send-thread], Controller 1's connection to broker id:1,host:tm1-kafkabroker101,port:9092 was unsuccessful (kafka.controller.RequestSendThread)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-04-16 10:48:48.276-0600] ERROR [Controller-1-to-broker-1-send-thread], Controller 1 epoch 38 failed to send LeaderAndIsr request with correlation id 1766 to broker id:1,host:tm1-kafkabroker101,port:9092. Reconnecting to broker. (kafka.controller.RequestSendThread)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-04-16 10:48:48.277-0600] ERROR [Controller-1-to-broker-1-send-thread], Controller 1's connection to broker id:1,host:tm1-kafkabroker101,port:9092 was unsuccessful (kafka.controller.RequestSendThread)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
It stopped our upgrade because of this issue.
Thanks,
Balaji