Hello Neha,
I am trying to run some tests which use Kafka 0.8.1.1. The tests do not
fail but give out a warning messages which I am trying to get rid off such
as :
2014-09-25 11:43:03,572 [kafka-processor-56598-1] ERROR
kafka.network.Processor - Closing socket for /127.0.0.1 because of error
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:67)
at kafka.network.MultiSend.writeTo(Transmission.scala:101)
at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:124)
at kafka.network.MultiSend.writeTo(Transmission.scala:101)
at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:219)
at kafka.network.Processor.write(SocketServer.scala:375)
at kafka.network.Processor.run(SocketServer.scala:247)
at java.lang.Thread.run(Thread.java:744)
Also,
2014-09-25 11:43:53,770 [kafka-processor-56598-1] ERROR
kafka.network.Processor - Closing socket for /127.0.0.1 because of error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:375)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:347)
at kafka.network.Processor.run(SocketServer.scala:245)
at java.lang.Thread.run(Thread.java:744)
I tried to debug the producer and consumer I am using but it turns out this
exception is thrown when I try to close the producer or consumer. I tried a
couple of approaches as mentioned in a few threads I saw online :
1.) I tried to increase the fetch.wait.max.ms and socket.timeout.ms for the
consumer but that didn't work
2.) A couple of threads suggested firewall issues. Since I am running the
tests locally, firewall issues are irrelevant since the broker is not
remote.
3.) I tried to catch the IOException in the tests which threw the above
mentioned exceptions but I was not able to catch them. This led me to
believe that the exceptions are not occurring on the producer and/or
consumer but on the broker. I think the broker seems to be closing open or
stray connections, which are giving out those exceptions. I also confirmed
that all my producer and consumer tests are closing connections to the
broker as soon as the test is executed.
Also, in a comment on this thread you mentioned that this is an expected
exception. Does that mean there will be a fix out for this?
Does this issue refer to the fix?
[1] - https://issues.apache.org/jira/browse/KAFKA-270
Thanks a lot for your help.
I am trying to run some tests which use Kafka 0.8.1.1. The tests do not
fail but give out a warning messages which I am trying to get rid off such
as :
2014-09-25 11:43:03,572 [kafka-processor-56598-1] ERROR
kafka.network.Processor - Closing socket for /127.0.0.1 because of error
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:67)
at kafka.network.MultiSend.writeTo(Transmission.scala:101)
at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:124)
at kafka.network.MultiSend.writeTo(Transmission.scala:101)
at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:219)
at kafka.network.Processor.write(SocketServer.scala:375)
at kafka.network.Processor.run(SocketServer.scala:247)
at java.lang.Thread.run(Thread.java:744)
Also,
2014-09-25 11:43:53,770 [kafka-processor-56598-1] ERROR
kafka.network.Processor - Closing socket for /127.0.0.1 because of error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:375)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:347)
at kafka.network.Processor.run(SocketServer.scala:245)
at java.lang.Thread.run(Thread.java:744)
I tried to debug the producer and consumer I am using but it turns out this
exception is thrown when I try to close the producer or consumer. I tried a
couple of approaches as mentioned in a few threads I saw online :
1.) I tried to increase the fetch.wait.max.ms and socket.timeout.ms for the
consumer but that didn't work
2.) A couple of threads suggested firewall issues. Since I am running the
tests locally, firewall issues are irrelevant since the broker is not
remote.
3.) I tried to catch the IOException in the tests which threw the above
mentioned exceptions but I was not able to catch them. This led me to
believe that the exceptions are not occurring on the producer and/or
consumer but on the broker. I think the broker seems to be closing open or
stray connections, which are giving out those exceptions. I also confirmed
that all my producer and consumer tests are closing connections to the
broker as soon as the test is executed.
Also, in a comment on this thread you mentioned that this is an expected
exception. Does that mean there will be a fix out for this?
Does this issue refer to the fix?
[1] - https://issues.apache.org/jira/browse/KAFKA-270
Thanks a lot for your help.