Skip to content

Follower misreads the entry after one whose payload ends at the end of the leader's transmission buffer (term 0x80000000) #308

Closed

Description

@galvesribeiro

Package: DotNext.Net.Cluster 6.11.1 (.NET 10), TCP and custom transports (ProtocolStream)

Summary

When a log entry's payload ends exactly where the leader's transmission buffer ends, the follower reads the next entry's
metadata 4 bytes early: it applies that entry with a term of 2147483648 (0x80000000), or the stream falls apart and the
member is reported unavailable.

On the leader, ProtocolStream.WriteAsync sends a frame that fills the buffer with a non-final header, so a payload ending
at the buffer's end is followed by an empty final frame (header 0x80000000). On the follower, ReceivedLogEntries.WriteToAsync
(Server.LogEntryProducer.cs) first tries ProtocolStream.TryReadFrameData. That succeeds whenever the frame holds the whole
payload, final or not, and leaves frameSize == 0 with readState == FrameStarted; the entry is then marked consumed, so
MoveNextAsync doesn't call SkipAsync, and the empty final frame's header is read as the start of the next entry's metadata.

// ProtocolStream.ReadOperations.cs, TryReadFrameData
EndReadFrameHeader();
var writtenBuffer = WrittenBuffer;
if (frameSize >= length && writtenBuffer.Length >= length) // also taken for a full, non-final frame
{
    AdvanceReadCursor(length);
    frame = writtenBuffer.Slice(0, length);
    frameSize -= length;
    return true;
}

Why it matters

The follower's log silently diverges from the leader's: the entry is applied with a wrong term, or, when the misread
metadata's length is wrong too, the exchange fails (MemberUnavailableException on the leader). It depends only on where
an entry's payload ends in the leader's buffer, so any workload can hit it.

Reproduction

Three nodes on the TCP transport, TransmissionBlockSize 4096. F is cut off (behind a TCP proxy) while A and B commit two
entries, then let back in, so it receives both in one AppendEntries. For first-entry sizes from 3976 to 4116 bytes:

DotNext.Net.Cluster 6.11.1.0, TCP transport, TransmissionBlockSize 4096
size 4002: MISREAD. Leader sent entry 57 (term 0, 4002 bytes) and entry 58 (term 0, 100 bytes of 0xa3).
  F applied entry 57: term 0, 4002 bytes, first bytes A2A2A2A2A2A2A2A2
  F applied entry 58: term 2147483648, 100 bytes, first bytes A3A3A3A3A3A3A3A3

The same in every run. The smallest form is the transport test in the PR below, which sends two entries over TcpClient
and TcpServer (block size 400) for every first-entry size from 1 to 1000 bytes and checks what the server received;
on 6.11.1 it fails with MemberUnavailableException at the size that ends at the buffer's end.

Suggested fix

Take the payload in one go only from a final frame, so a full, non-final frame goes through the stream, which reads the
empty final frame too:

-                if (frameSize >= length && writtenBuffer.Length >= length)
+                if (readState is ReadState.EndOfStreamReached && frameSize >= length && writtenBuffer.Length >= length)

With it, the reproduction above finds no misread over the whole range, and the cluster tests pass.

Activity

  1. added theissue type on Nov 2, 2026
  2. added
    Lib:ClusterDotNext.Net.Cluster library
    ai_assistedBug or PR is produced partially or fully by the AI model (Codex, Claude, etc.)
    on Nov 2, 2026
  3. sakno commented on Nov 6, 2026

    @sakno
    Collaborator

    Fixed in 6.12.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Lib:ClusterDotNext.Net.Cluster libraryai_assistedBug or PR is produced partially or fully by the AI model (Codex, Claude, etc.)

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions