Skip to content

ProcessCommitted can lower LastAppliedIndex, re-applying entries after a restart #37

Open

Description

@resetius

Problem

TRequestProcessor::ProcessCommitted always ends with Rsm->LastAppliedIndex = commitIndex (raft.cpp#L631-L646), also when commitIndex is lower than the current LastAppliedIndex. The loop above it does nothing in that case, so the assignment moves the applied index backwards.

The SQL example loads LastAppliedIndex from its raft_metadata_ table at startup (sql.cpp#L74-L79), while TVolatileState::CommitIndex starts at 0 after a restart.

Scenario

  1. The SQL server applies entries 1..100; LastAppliedIndex = 100 is persisted in the database.
  2. The server restarts and loads LastAppliedIndex = 100. The Raft CommitIndex is 0.
  3. The first pass of ProcessCommitted runs with commitIndex = 0 and sets Rsm->LastAppliedIndex = 0.
  4. As soon as the node learns the real commit index (say 100), the loop re-applies entries 1..100. TSql::Write checks LastAppliedIndex < index (sql.cpp#L155), which is now true again, so non-idempotent statements run a second time.

Expected

lastApplied increases monotonically (Raft paper, Figure 2, "All Servers"), and an entry is never applied twice to a state machine that has already applied it: ProcessCommitted must only raise LastAppliedIndex, and the progress recovered from the state machine must be honoured after a restart.

Fix in PR #31

PR #31 (closed, not merged) has the fix in a0c593d (ProcessCommitted never lowers LastAppliedIndex; recovered progress protects the committed prefix). Tests that cover it: test_recovered_application_progress_is_not_rewound, test_application_progress_never_decreases_on_lower_commit_input. Since the PR is closed, the fix would come as a separate small PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions