Repository navigation
Conversation
* Reject corrupt persistence on recovery and fix log truncation state * Fail-stop on live storage errors instead of escaping into network handlers * Cover replacement after live truncation without reopening storage
* Add Linux durable state with atomic publication and crash-boundary tests * Verify Raft election and application replay across durable-state restart
… leadership term (#4)
…tions and casts (#7)
* Preserve snapshot chunk identity across retries to avoid ACK starvation * Protect active reconnects from stale reader failures
* Pin active snapshot transfers across ongoing compaction * Avoid competing connect deadline wakeups after completion
Add runtime learner target management to TRaftServer, connect staged targets to server draining, and document endpoint lifecycle and remaining membership limits.
Check the embedded boundary, declared size, and voter list before a covered InstallSnapshot fast-path acknowledgement.
|
Thank you for this work: snapshots, log compaction, InstallSnapshot, learners and membership changes are exactly what this repository needs, and the PR comes with a lot of careful testing. It was closed within a minute, and the same branch then appeared as PR 47 in the fork, so it was probably opened here by accident. I still went through it, because a lot of it would be welcome upstream. Below is a summary and a list of things to adjust when the work is split into smaller PRs. A proposed split plan follows in a separate comment, and there are inline notes on the key lines. What makes an implementation a good fit hereAny implementation of these features is welcome if it stays generic:
What the PR containsThe title describes the last commit (
There is no generated or vendored code, and the coroio submodule pointer is unchanged. CI. The first runs never started: first-time-contributor approval was still pending when the PR was closed, so both runs show "failure" with 0 jobs; the workflow files are fine. I re-ran both on the merge commit with current
Bugs in
|
Proposed split into PRsEvery PR targets
Fork commits are referred to by short SHA. "Parallel" means a PR does not wait on the others in its stage. Stage A: independent fixes for
|
resetius
left a comment
There was a problem hiding this comment.
Inline notes for the summary comment above, ranked P1/P2/P3 the same way. The "Good" notes mark code worth bringing upstream first; the numbers refer to the split-plan comment.
| // still be registered with the poller; never destroy that live coroutine | ||
| // because an older reader failed. Established connectors are done here. | ||
| Connected = false; | ||
| if (Drainer) { Drainer.destroy(); Drainer = {}; } |
There was a problem hiding this comment.
P1 (possible use-after-free; found by reading the code, not reproduced). Disconnect runs inside OutboundServe and can destroy Drainer while it is suspended in TMessageWriter::Write on the same socket, for example when the peer stopped reading and the send buffer is full.
When the peer then resets the connection, epoll/kqueue report read and write readiness in the same batch. coroio queues both handles in ReadyEvents_ (READ first) and resumes them one by one:
- The reader fails and calls
Disconnect. Drainer.destroy()frees the writer frames (through theTValueTaskdestructors).WakeupReadyHandlesthen resumes the queued WRITE handle of a destroyed frame.
The RemoveEvent queued by the socket destructor only takes effect at the next poll, so it doesn't help here.
The guard above already protects a live Connector for this reason. Treating a suspended Drainer the same way would close the gap: mark the node disconnected and let the writer fail and unwind on its own, or defer destruction until it is done.
| State->InstallSnapshot(snapshot); // durable snapshot and suffix before application replacement or ACK | ||
| application->LoadSnapshot(snapshot.Data, snapshot.Index); | ||
| if (application->LastAppliedIndex != snapshot.Index) std::terminate(); | ||
| } catch (...) { std::terminate(); } |
There was a problem hiding this comment.
P2. A valid snapshot from a legitimate leader can end here in std::terminate(). DecodeSnapshot accepts MRFTS003 and the checks above pass. However, TDurableState::InstallSnapshot (and TState::InstallSnapshot) throw "Configuration snapshots require a bound voter set" when the store has no voter binding. The TRaft constructor leaves a non-fresh store without a snapshot and without learners unbound; an MRFTD001 store is one example. After a restart the node is still behind, so it would terminate again.
Suggestion: check every precondition that depends only on local state (binding, index/term ordering) before this block and reply Success = 0. Keep this path for genuine I/O failures, reported to the embedder.
| if (snapshot.StableConfiguration && | ||
| (snapshot.ConfigurationIndex != knownSnapshot->ConfigurationIndex || | ||
| snapshot.ConfigurationTerm != knownSnapshot->ConfigurationTerm)) | ||
| throw TMessageError("Stable snapshot configuration identity changed"); |
There was a problem hiding this comment.
P2 (liveness). When the follower rejects an image here (a conflicting identity, or ProbeCoveredSnapshot returning Supported = false on the covered path), the leader resets the transfer to offset 0 (lines 923-925) and sets RpcDue to 0 (line 939). It then streams the whole image again; nothing changes on either side, so this repeats forever without backoff or a diagnostic.
Suggestion: if the condition can't happen in a correct cluster, an assert is enough. Otherwise, give it a terminal outcome: stop the transfer to that peer and report the peer to the embedder. A test that injects a permanently rejecting follower would pin this down.
| if (!snapshot.StableConfiguration) { | ||
| // A legacy-format refresh has no configuration identity. Keep | ||
| // the receiver's trusted identity when storing the new image. | ||
| snapshot.StableConfiguration = true; |
There was a problem hiding this comment.
P2 (design). Here the receiver writes its own ConfigurationIndex/ConfigurationTerm into the image the leader sent. Raft has a simpler rule: a snapshot carries the configuration in effect at its last included index, taken from the sender's log (Ongaro & Ousterhout 2014, §7 and Fig. 13; thesis ch. 5).
Once configuration changes are entries in the log, every node derives the same configuration for the same index. The leader's snapshot then already carries the right configuration, and nothing needs merging or rejecting on the receiving side. For the split, it would be easiest to land this together with membership changes (19 in the split plan) and to drop the separate stable-identity format.
| // replication progress. Stable metadata identity is checked as well. The | ||
| // application payload and CRC are unnecessary because the receiver already | ||
| // has this committed log prefix. | ||
| const size_t fixedHeaderBytes = stable ? 52 : 36; |
There was a problem hiding this comment.
P3. This is a second, hand-written header parser (offsets 8/16/24/28/36/44, header sizes 36/52) that has to stay in sync with EncodeSnapshot/DecodeSnapshot in snapshot.cpp. The covered case doesn't need the image at all: when SnapshotIndex <= commitIndex and the terms match, the follower already holds every entry the snapshot covers (Log Matching), so it can acknowledge right away without buffering. That would remove this parser and the extra IncomingSnapshot state for the covered path.
| auto commitIndex = std::max(CommitIndex, Indices[nservers / 2]); | ||
| // Ascending order: a strict majority must have replicated this index. | ||
| // For even cluster sizes, the upper median would accept only half. | ||
| auto commitIndex = std::max(CommitIndex, Indices[(nservers - 1) / 2]); |
There was a problem hiding this comment.
Good: a real bug in master. With an ascending sort, Indices[nservers / 2] lets a 4-node cluster commit with 2 of 4 replicas. This line with test_majority_all_cluster_sizes would be a great first standalone PR (1 in the split plan).
| continue; | ||
| } | ||
| if (Waiting.front().Index > lastApplied) break; | ||
| auto w = Waiting.front(); Waiting.pop(); |
There was a problem hiding this comment.
Good: also a master bug. The old loop checked and replied to Waiting.back() but popped the front, so when several requests completed at once, the newest got several replies and the older ones got none. Front/pop for both queues, with the FIFO tests, makes a small independent PR (2 in the split plan).
| return VolatileState->CommitSeqno; | ||
| if (StateName != EState::LEADER || VolatileState->CommitIndex == 0 || | ||
| State->LogTerm(VolatileState->CommitIndex) != State->CurrentTerm) return 0; | ||
| return VotingQuorum->Value([&](uint32_t id) -> uint64_t { |
There was a problem hiding this comment.
Good. This is the ReadIndex rule from the thesis (§6.4): a commit in the current term, plus a majority acknowledging RPCs sent after the read barrier. In master, one follower's reply is enough to advance CommitSeqno. The barrier, the per-peer ReadAckSeqno and this quorum would make a focused fix (3 in the split plan).
|
|
||
| // In particular, after truncation to zero the stream may still point into | ||
| // the old suffix. Always position the output explicitly. | ||
| Entries.seekp(offset); |
There was a problem hiding this comment.
Good: a real master bug. After the log was truncated to zero, Append recorded offset 0 but wrote wherever the stream position had been left. This explicit seekp, together with refusing to reinitialise a store of unexpected size (line 32), would make a nice small TDiskState PR (6 in the split plan). Errors there could surface as exceptions instead of Fail/std::terminate.
| NNet::TValueTask<TMessageHolder<TMessage>> ReadFrame(bool logOnly, uint64_t& remaining) { | ||
| TMessage header; | ||
| co_await NNet::TByteReader(Socket).Read(&header, sizeof(header)); | ||
| ValidateMessageHeader(header.Type, header.Len, Limits); |
There was a problem hiding this comment.
Good. Validating type and length before allocating fixes master's unbounded NewHoldedMessage(type, len) and the len - sizeof(TMessage) underflow. The reader part alone would be a small standalone PR (7 in the split plan), with the batch limits following later.
Summary
Validation
git diff --checkpassed.This is a self-reviewed patch; no independent reviewer approval is claimed.