Skip to content

TDiskState forgets term and vote on unexpected state size; Append offset wrong after truncating to zero #38

Open

Description

@resetius

Problem

Two defects in TDiskState (persist.cpp):

  1. Silent reinitialisation. The constructor reads the state file only if its size is exactly sizeof(LastLogIndex) + sizeof(CurrentTerm) + sizeof(VotedFor); for any other size (truncated, corrupt, or from another format) it calls Commit() and overwrites the file with the zero-initialised fields (persist.cpp#L10-L19). The current term, the vote, and the log length are forgotten without any error, while the entries and index files still hold data.
  2. Append after truncation to zero. Append sets offset = 0 when Get(LastLogIndex) returns nothing and then writes the entry with Write at the current put position of Entries (persist.cpp#L72-L85). RemoveLast only decrements LastLogIndex (persist.cpp#L64-L70), so after the log has been truncated to zero the stream position is wherever the last reads or writes left it. The index file then says "offset 0", and the entry is somewhere else.

Scenario

  1. A node stores a vote for term 7, then crashes in the middle of the state-file write. On restart the file has an unexpected size, the node starts with term 0 and no vote, and can vote again in term 7 for a different candidate: two leaders in one term are possible (Raft paper section 5.2, "each server will vote for at most one candidate in a given term").
  2. A follower's whole log is removed by conflict resolution (RemoveLast down to 0), then it appends a new first entry. The entry bytes land at the old stream position, but the index says offset 0, so Get(1) reads a different (stale) entry or garbage.

Expected

A store of unexpected contents is reported as an error (an exception to the embedder) instead of being reinitialised; CurrentTerm, VotedFor and the log are persistent state that must survive restarts (Raft paper Figure 2). Append seeks explicitly to the end (or to the recorded offset) before writing, so the recorded offset always matches where the bytes are.

Fix in PR #31

PR #31 (closed, not merged) has the fixes in 781ce32 and 93f8332 (no silent reinitialisation, explicit seekp in Append, offset and term validation on load; failures are reported as exceptions). Tests that cover it: test/test_persist_recovery.cpp and the test_disk_state* adjustments. Since the PR is closed, the fix would come as a separate small PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions