Quantcast
Channel: Kafka Timeline
Viewing all articles
Browse latest Browse all 1519

Different free space and log files between nodes

$
0
0
Hi,

I've noticed an interesting behaviour which I hope someone can fully
explain.
I have 3 Kafka Node cluster with a setting of log.retention.hours=168 (7
days) and log.segment.bytes=536870912.

I recently restarted one of the nodes and it's uptime is now 3 days behind
than the other 2.

After about 7 days I noticed that the other 2 nodes cleared out an equal
amount of stale logs/data,
but the restarted node didn't cleared out the same amount. The restarted
node only cleared out a
similar amount 3 days later. Generally now the restarted node seems to be 3
days (free space wise) behind
the other 2.

I noticed that certain partition log and index files on the restarted node
are different to the other 2.
Below is an example (Node C is the restarted one)

Node A - Topic Z Partition 12
[ 488 Sep 26 11:47] 00000000000000046460.index
[ 1781829 Sep 24 12:09] 00000000000000046460.log
[ 10485760 Sep 28 22:23] 00000000000000046522.index
[ 1536693 Sep 28 22:23] 00000000000000046522.log

Node B - Topic Z Partition 12
[ 488 Sep 26 11:47] 00000000000000046460.index
[ 1781829 Sep 24 12:09] 00000000000000046460.log
[ 10485760 Sep 28 22:23] 00000000000000046522.index
[ 1536693 Sep 28 22:23] 00000000000000046522.log

Node C - Topic Z Partition 12
[ 10485760 Sep 28 22:23] 00000000000000046485.index
[ 2277311 Sep 28 22:23] 00000000000000046485.log

I can see that Node C's base offset (log prefix) is in between the offsets
of the other logs on Node A B, and that
suggests to me some partition 12 messages are on Node A B but not on Node
C?

I was hoping someone could help me figure out what's happening.

Thanks
Dayo

Viewing all articles
Browse latest Browse all 1519

Trending Articles