cannot replicate topics kafka inconsistent state

Hi,

I've installed kafka 2.8.1,
created a topic using:

/opt/kafka/bin/kafka-topics.sh --create --topic "test" --zookeeper
"localhost:2381" --partitions 2 --replication-factor 2

Then opened a console producer and a console consumer.
I type a few lines on the producer and then the two kafka brokers that
should have the two replicas start throwing errors to the logs, the only
way to get kafka back to normal again is by deleting all of the topic data
in kafka and in zookeeper and restarting.

The errors are:
broker1:

2014-06-17/01:40:32.137/PDT ERROR [kafka-processor-9092-5]:
kafka.network.Processor - Closing socket for /10.101.4.218 because of
error^C

kafka.common.KafkaException: This operation cannot be completed on a
complete request.

at kafka.network.Transmission$class.expectIncomplete(Transmission.scala:34)

at kafka.api.FetchResponseSend.expectIncomplete(FetchResponse.scala:191)

at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:214)

at kafka.network.Processor.write(SocketServer.scala:375)

at kafka.network.Processor.run(SocketServer.scala:247)

at java.lang.Thread.run(Thread.java:744)

broker2

2014-06-17/01:40:29.127/PDT WARN [ReplicaFetcherThread-0-215]:
kafka.consumer.SimpleConsumer - Reconnect due to socket error: null

2014-06-17/01:40:29.127/PDT ERROR [ReplicaFetcherThread-0-215]:
kafka.server.ReplicaFetcherThread - [ReplicaFetcherThread-0-215], Error in
fetch Name: FetchRequest; Version: 0; CorrelationId: 545271; ClientId:
ReplicaFetcherThread-0-215; ReplicaId: 218; MaxWait: 1000 ms; MinBytes: 1
bytes; RequestInfo: [test,1] -> PartitionFetchInfo(1,2147483647)

java.io.EOFException: Received -1 when reading from channel, socket has
likely been closed.

at kafka.utils.Utils$.read(Utils.scala:376)

at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)

at kafka.network.Receive$class.readCompletely(Transmission.scala:56)

at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)

at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)

at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:81)

at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71)

at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109)

at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)

at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109)

at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)

at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108)

at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)

at
kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108)

at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)

at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:107)

at
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96)

at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)

The same error on each is repeated in an endless loop.

My config server config is:

num.network.threads=24

num.io.threads=24

socket.send.buffer.bytes=10485760

socket.receive.buffer.bytes=1048576

socket.request.max.bytes=524288000

replica.lag.max.messages=5000000

replica.fetch.max.bytes=2097152

replica.fetch.wait.max.ms=1000

log.dir=/data

num.partitions=12

log.flush.interval.messages=200000

log.flush.interval.ms=2000

log.retention.hours=168

log.retention.mins=10080

log.retention.hours=168

log.retention.mins=10080

log.retention.bytes=2199023255552

replica.fetch.max.bytes=2147483647

log.segment.bytes=209715200

log.cleanup.interval.mins=10

default.replication.factor=2

zookeeper.connect=localhost:2381

zookeeper.connection.timeout.ms=1000000

Am I missing some configuration properties?

Regards,

Gerrit