Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
76aae7a
Fix quorum and request ordering; establish deterministic correctness …
mrayva Nov 8, 2026
781ce32
Reject corrupt storage on recovery and fix truncation state (#2)
mrayva Nov 8, 2026
4e706a4
Add opt-in durable Linux storage with atomic image publication (#3)
mrayva Nov 8, 2026
d114d5d
Require fresh majority confirmation for strong reads and bind them to…
mrayva Nov 8, 2026
e117c18
Bind pending writes to leadership terms and clean up request ownershi…
mrayva Nov 8, 2026
eee8ddc
Preserve forwarded client cookies and validate reply routing context …
mrayva Nov 8, 2026
b36e5b1
Bound network frames and replication batches; validate message alloca…
mrayva Nov 8, 2026
cbbbfa1
Validate consensus RPCs and correlate bounded replication acknowledge…
mrayva Nov 8, 2026
ca86106
Validate application payloads before admission and bound RSM data acc…
mrayva Nov 8, 2026
b514088
Enforce read-only SQL execution before stepping statements (#10)
mrayva Nov 8, 2026
5849d85
Keep SQL application changes and progress within managed transactions…
mrayva Nov 8, 2026
f958c43
Bound SQL read execution and serialize replies within a byte budget (…
mrayva Nov 8, 2026
a0c593d
Preserve recovered application progress and protect its committed pre…
mrayva Nov 8, 2026
93f8332
Reject impossible log terms in legacy storage recovery and reads (#14)
mrayva Nov 9, 2026
4b76d8b
Add durable KV checkpoints with validated snapshot recovery (#15)
mrayva Nov 9, 2026
edea429
Compact durable logs and install chunked snapshots on lagging followe…
mrayva Nov 9, 2026
9a5752e
Add non-voting learners and persist fixed voter configuration (#17)
mrayva Nov 9, 2026
049907d
Fix snapshot catch-up starvation and reconnect use-after-free (#18)
mrayva Nov 9, 2026
eb16ab6
Fix snapshot compaction starvation and unsafe connect deadlines (#19)
mrayva Nov 9, 2026
e0a4cb9
Reject nonleader proposals before mutating the Raft log (#20)
mrayva Nov 13, 2026
704ae08
Isolate in-memory log entries from mutable callers (#21)
mrayva Nov 13, 2026
6d9c758
Avoid copying log payloads for term checks (#22)
mrayva Nov 13, 2026
afde6fa
Validate message buffer and batch extents before access (#23)
mrayva Nov 13, 2026
452377c
Add joint quorum arithmetic foundation (#24)
mrayva Nov 13, 2026
5ae2203
Add joint snapshot metadata with fail-closed runtime guards (#25)
mrayva Nov 13, 2026
db5794b
Add fail-closed configuration entry foundation (#26)
mrayva Nov 13, 2026
99efd7d
Validate offline configuration history and snapshot identities (#27)
mrayva Nov 13, 2026
f510e99
Validate configuration history across retained logs (#28)
mrayva Nov 13, 2026
b20fba6
Reject snapshot terms contradicting committed history (#29)
mrayva Nov 14, 2026
d202c36
Reject snapshot acknowledgements that skip bytes (#30)
mrayva Nov 14, 2026
ff79596
Reject snapshot storage boundary term regression (#31)
mrayva Nov 14, 2026
005b14a
Reject storage appends beyond maximum log index (#32)
mrayva Nov 14, 2026
326033e
Reject covered snapshots with impossible terms (#33)
mrayva Nov 14, 2026
47218e8
Add offline stable configuration snapshot metadata (#34)
mrayva Nov 14, 2026
467bb14
Project offline configuration history into snapshots (#35)
mrayva Nov 14, 2026
42fb6c0
Validate snapshot boundaries before Raft recovery (#36)
mrayva Nov 14, 2026
91d8a0c
Validate compacted suffix terms during recovery (#37)
mrayva Nov 14, 2026
3126493
Validate recovered votes against fixed voters (#38)
mrayva Nov 14, 2026
25890fe
Support leader-local online learner staging (#39)
mrayva Nov 14, 2026
c912788
Integrate staged learners with the network server (#40)
mrayva Nov 14, 2026
3010108
Validate covered snapshot identity before ACK (#41)
mrayva Nov 14, 2026
53e1a0e
Expire idle partial snapshot transfers (#42)
mrayva Nov 14, 2026
c52eb0a
Bound pinned snapshot generations during compaction (#43)
mrayva Nov 14, 2026
06d936a
Add SQL snapshot capture and recovery (#44)
mrayva Nov 14, 2026
0fea03f
Persist bound configuration snapshots for offline recovery (#45)
mrayva Nov 14, 2026
a2cd136
Enable fixed-membership stable snapshot recovery (#46)
mrayva Nov 14, 2026
2463892
Preserve stable snapshot identity across refreshes
mrayva Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/correctness.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: Consensus correctness
on: [push, pull_request]
permissions:
contents: read
jobs:
core:
runs-on: ubuntu-latest
strategy:
matrix:
build_type: [Debug, Release]
steps:
- uses: actions/checkout@v4
- run: sudo apt-get update && sudo apt-get install -y libcmocka-dev libsqlite3-dev
- name: Configure core without networking dependencies
run: >-
cmake -S . -B build/core
-DMINIRAFT_BUILD_NETWORK=OFF
-DCMAKE_BUILD_TYPE=${{ matrix.build_type }}
- run: cmake --build build/core --parallel 2
- run: ctest --test-dir build/core --output-on-failure
sanitizers:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- run: sudo apt-get update && sudo apt-get install -y libcmocka-dev liburing-dev libssl-dev libsqlite3-dev
- name: Configure full build with ASan and UBSan
run: >-
cmake -S . -B build/full -DCMAKE_BUILD_TYPE=Debug
-DCMAKE_CXX_FLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all"
-DCMAKE_EXE_LINKER_FLAGS="-fsanitize=address,undefined"
- run: cmake --build build/full --parallel 2
- run: ctest --test-dir build/full --output-on-failure
2 changes: 1 addition & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[submodule "coroio"]
path = coroio
url = ../coroio.git
url = https://github.com/resetius/coroio.git
15 changes: 10 additions & 5 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,15 @@ if (EXISTS ${CMAKE_BINARY_DIR}/local.cmake)
include(${CMAKE_BINARY_DIR}/local.cmake)
endif ()

enable_testing()
include(CTest)
option(MINIRAFT_BUILD_NETWORK "Build coroio networking and examples" ON)

add_subdirectory(coroio/coroio)
add_subdirectory(miniraft)
add_subdirectory(miniraft/net)
add_subdirectory(examples)
add_subdirectory(test)
if(MINIRAFT_BUILD_NETWORK)
add_subdirectory(coroio/coroio)
add_subdirectory(miniraft/net)
add_subdirectory(examples)
endif()
if(BUILD_TESTING)
add_subdirectory(test)
endif()
98 changes: 96 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,99 @@
# MiniRaft-CPP

This fork's first milestone fixes strict-majority commit selection and FIFO
request completion, and adds deterministic consensus failure tests.
See the [correctness baseline and review](docs/correctness-baseline.md) for
validation, build instructions and remaining production release gates.
The milestone is experimental; durable crash recovery is not established.
The [persistence recovery follow-up](docs/persistence-recovery.md) rejects
incomplete or structurally corrupt storage and fixes truncation/restart behavior.
For new Linux stores, the opt-in [durable state backend](docs/durable-state.md)
adds checksummed atomic publication with file and directory synchronization.
The KV example selects it with `--durable`; legacy `--persist` is unchanged.
The [strong-read follow-up](docs/strong-reads.md) requires a fresh majority for
`--consistent` reads and rejects pending reads when their leadership term ends.
The [pending request follow-up](docs/pending-requests.md) prevents replacement
write acknowledgements and cleans up obsolete or disconnected client requests.
The [forwarding follow-up](docs/forwarding.md) preserves client cookies and binds
forwarded replies to their original leader, term and outbound peer.
The [network framing follow-up](docs/network-frames.md) handles fragmented input
and enforces type, frame, batch and AppendEntries payload bounds.
The [consensus validation follow-up](docs/consensus-validation.md) checks RPC
identities and indices, protects committed prefixes and correlates acknowledgements.
The [application payload follow-up](docs/application-payloads.md) validates KV,
SQL and dummy-RSM data before access and provides pre-admission validation hooks.
The [SQL read policy](docs/sql-readonly.md) prevents requests marked as reads
from executing unreplicated mutations or connection controls.
The [SQL application follow-up](docs/sql-application.md) protects managed write
transactions and commits application progress together with successful changes.
The [SQL read budget follow-up](docs/sql-read-budgets.md) interrupts expensive
reads and bounds serialized output before appending each field or separator.
The [application recovery follow-up](docs/application-recovery.md) preserves
recovered progress and prevents committed commands from being applied again.
The [legacy log-term follow-up](docs/legacy-log-terms.md) rejects impossible
stored term sequences before they influence elections or log matching.
The [snapshot foundation](docs/snapshots.md) adds opt-in durable KV checkpoints
and tail-only application recovery with `--durable --snapshots N`. Logs remain
fully retained in that baseline. The [compaction follow-up](docs/compaction.md)
enables prefix deletion and chunked InstallSnapshot catch-up; fixed membership
remains required. Upgrade all voters before enabling the new compaction behavior.
The [learner follow-up](docs/learners.md) adds non-voting catch-up and durable
voter binding. KV uses `--learner id` on every replica's configuration. Promotion
and joint-consensus voter changes remain disabled.
The [snapshot retry follow-up](docs/snapshot-retries.md) prevents timeout retries
and strong-read traffic from starving delayed snapshot acknowledgements, and
prevents stale reader failures from destroying newer reconnect attempts.
The [snapshot pinning follow-up](docs/snapshot-pinning.md) lets an active transfer
finish while the leader continues compacting newer application state, and avoids
the dependency's unsafe connect-deadline path. TCP connect failure timing follows
the OS; a safe bounded application deadline remains open.
The [proposal admission follow-up](docs/proposal-admission.md) restricts direct
Append calls to leaders, preventing nonleaders from creating conflicting commands
at the same log index and term.
The [memory log ownership follow-up](docs/memory-log-ownership.md) prevents
caller-owned entry buffers from changing entries already stored in TState.
The [term lookup follow-up](docs/term-lookups.md) avoids copying full payloads
when Raft needs only a retained entry's term.
The [joint quorum foundation](docs/joint-quorum.md) adds and tests dual-majority
arithmetic for future membership changes; live voter reconfiguration remains disabled.
The [joint snapshot metadata foundation](docs/joint-snapshots.md) adds a versioned
representation for a joint configuration at a snapshot boundary; runtime adoption
and membership changes remain disabled.
The [configuration entry foundation](docs/configuration-entries.md) defines an
experimental entry codec and rejects that entry kind in the fixed-membership runtime.
The [configuration history validator](docs/configuration-history.md) checks an
offline retained-log sequence against a trusted baseline or joint snapshot.
The [configuration log scanner](docs/configuration-scan.md) validates a retained
suffix at a caller-supplied commit boundary and at its end.
The [online learner staging API](docs/online-learner-staging.md) lets embedders
add and remove leader-local, non-voting replication targets.
The [network learner follow-up](docs/online-learner-network.md) connects that
API to the built-in server's runtime drain loop.
The [covered-snapshot follow-up](docs/covered-snapshot-identity.md) validates
snapshot boundary and voter metadata before acknowledging an already-covered image.
The [snapshot transfer expiry follow-up](docs/snapshot-transfer-expiry.md)
releases incomplete incoming images after 30 seconds without a valid chunk,
including on learners.
The [snapshot pin budget follow-up](docs/snapshot-pin-budget.md) bounds outgoing
transfer retention to two image generations and defers local compaction at the cap.
The [stable configuration snapshot foundation](docs/stable-configuration-snapshots.md)
preserves configuration identity. Raft can recover and install stable metadata
snapshots when their voter list matches the node's existing fixed membership;
joint snapshots and live voter changes remain disabled.
The [durable configuration snapshot follow-up](docs/durable-configuration-snapshots.md)
documents its voter-binding requirement and recovery boundary.
The [stable snapshot runtime follow-up](docs/stable-configuration-snapshot-runtime.md)
carries that identity through restore, InstallSnapshot, and later compaction.
The [metadata continuity follow-up](docs/stable-snapshot-metadata-continuity.md)
preserves the known identity across legacy-format refreshes and rejects conflicts.
The [snapshot term validation fix](docs/snapshot-committed-terms.md) rejects
snapshot boundaries that contradict the follower's committed prefix.
The [message buffer follow-up](docs/message-extents.md) checks allocated buffer
and batch extents before accepting locally constructed frames.
The [SQL snapshot follow-up](docs/sql-snapshots.md) adds bounded SQLite snapshot
capture, validated restore and recovery from an application checkpoint left
behind a durably installed Raft snapshot.

[![Number of GitHub stars](https://img.shields.io/github/stars/resetius/miniraft-cpp)](https://github.com/resetius/miniraft-cpp/stargazers)
![GitHub commit activity](https://img.shields.io/github/commit-activity/m/resetius/miniraft-cpp)
[![GitHub license which is BSD-2-Clause license](https://img.shields.io/github/license/resetius/miniraft-cpp)](https://github.com/resetius/miniraft-cpp)
Expand Down Expand Up @@ -31,8 +125,8 @@ log replication, and fault tolerance in a distributed system.
- **Leader Election** (supported)
- **Log Replication** (supported)
- **Persistence** (supported)
- **Snapshots** (not supported yet)
- **Membership Change** (not supported yet)
- **Snapshots** (opt-in KV compaction and InstallSnapshot; Linux durable backend)
- **Membership Change** (non-voting learners; voting changes not supported yet)

The project focuses on a clear, modular design that leverages C++20 coroutines to simplify asynchronous flows.

Expand Down
49 changes: 49 additions & 0 deletions docs/application-payloads.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Application payload validation: v0.1.0-alpha.9

KV key/value lengths and SQL query lengths are checked against the actual
remaining bytes before preparing, reading or applying a command. Checks use the
payload's offset rather than sizeof(derived structure), preserving legacy native
layout/tail padding. Invalid direct application writes do not advance application
progress or mutate KV state.

IRsm gains a default no-op const Validate hook. TRequestProcessor calls it before
queueing, preparing or forwarding requests. KV, SQL and the dummy RSM implement
their shape checks. Custom applications must supply their own validator: the
default preserves source compatibility but cannot infer custom payload formats.
Admission validation exceptions propagate to the inbound handler, closing that
client connection before the invalid operation is admitted.

Read-time standard exceptions are returned as ErrorCode=2 with the original
cookie, allowing other requests and timer processing to continue. Exceptions
while applying a committed entry terminate the process: continuing after a
partial or invalid committed application is unsafe. This is an intentional
behavioral change for embedders whose Write callback throws.

The dummy RSM requires a complete read index and copies only the log entry's
data into its reply, fixing a header-size error that overread by 16 bytes.
Message factories take size_t allocation sizes and reject values above UINT32_MAX
before narrowing/allocation. KV list and SQL reply construction enforce the
existing frame cap. Blank/malformed KV CLI commands no longer dereference missing
tokens or attempt to send an empty request.

KV/SQL state machines are split from their CLI drivers so the real application
methods are directly testable without networking. SQL tests use SQLite's genuine
:memory: database; file-backed naming remains unchanged. SQLite development files
are installed in the core CI job. SQL tests are optional when SQLite is absent.

Fifteen new tests cover malformed lengths, valid KV/SQL round trips, rejection
before admission/forwarding, no mutation/progress on rejection, oversized factory
arguments, dummy read bounds, recoverable read failures and fatal committed
application failures. All 145 tests pass locally on Linux with SQLite under
ASan/UBSan; core Release checks pass. Hosted CI executes the same targets.
The fatal-application subprocess test is excluded on Windows.

Self-review checked validation before data access and admission, widening before
length addition, native-layout offsets, response allocation bounds, CLI tokens
and exception routing. No independent reviewer approval is claimed.

This is payload memory-safety validation, not SQL authorization or complete state
machine correctness. SQL read-only enforcement, nondeterministic SQL, transaction
and metadata semantics, result-set memory quotas, authentication, deadlines,
admission limits and retry deduplication remain open. Callback access remains
serialized and non-reentrant. Write outcomes after interruption can be unknown.
67 changes: 67 additions & 0 deletions docs/application-recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Recovered application progress: v0.1.0-alpha.13

The [compaction follow-up](compaction.md) extends this recovery contract to
embedded snapshots and absolute log boundaries.

After restarting a durable application, Raft's volatile commit index began at
zero. ProcessCommitted unconditionally copied that zero into LastAppliedIndex,
discarding the application's recovered marker. Re-establishing commitment then
reapplied old commands. A file-backed SQL/TDiskState regression reproduced an
INSERT being applied twice after a clean restart. A separate restart scenario
also exposed the loss of committed-prefix protection before leader contact.

TRequestProcessor construction now initializes a commit floor from the recovered
application marker. The marker must not exceed the recovered local log or the
signed log-index range; invalid pairings throw before changing the commit index.
The floor preserves any higher commitment already known to Raft. The existing
AppendEntries validation then protects this applied prefix immediately, while
allowing replacement of an uncommitted suffix. The restoration helper is private
to Raft and the request processor.

ProcessCommitted returns when commitment is at or below application progress.
It never rewinds the marker or computes LastAppliedIndex+1 in that case. New
commitment applies only the remaining entries. Framework progress still advances
over internal no-op entries; an application need only persist progress alongside
its own mutations, so harmless no-ops can be revisited after restart.

Recovery contract for embedders:

* Restore application state and LastAppliedIndex atomically before constructing
TRequestProcessor, and construct the processor before accepting network traffic.
* A nonzero marker must represent genuinely committed progress in the same local
Raft log. Do not derive it from the log's last index: a suffix may be uncommitted.
* A volatile application starts at zero and rebuilds after commitment is known.
* Keep application and Raft storage paired. An application ahead of the local log
now fails startup instead of silently forgetting or skipping progress.

An applied marker is trusted evidence of commitment; a range check cannot detect
a forged marker, mismatched data at the same index or application state persisted
without its marker. Snapshot/compaction recovery is not supported. Strong reads
still require a current-term commit and a fresh majority; the recovered floor
does not replace those checks. This follows the monotonic application-progress
and state-machine safety rules in [Raft's Figure 2 and Figure 3](https://raft.github.io/raft.pdf).

Six new tests failed against the prior implementation and now pass:

1. Restarted application progress survives an early timer and is not replayed.
2. Lower commitment input cannot rewind application progress.
3. A recovered applied prefix resists conflicting RPCs while its suffix can change.
4. Application progress ahead of the recovered log is rejected without mutation.
5. A three-node restart preserves durable application histories, rebuilds a
volatile application, replaces an uncommitted old-leader suffix and tolerates
delayed, reversed and duplicate traffic after a new election. It also checks
that recovered commitment cannot authorize new-term reads by itself.
6. Real SQL and legacy Raft files reopen without repeating a committed INSERT.

All 175 tests across 11 CTest targets pass locally under ASan/UBSan with SQLite;
the Release core suite also passes. Self-review covered initialization ordering,
range checks before mutation, monotonicity, no-op handling, conflict protection,
current-term read gating and repeat application calls. No independent reviewer
approval is claimed.

The simulator retains stable term/vote/log objects and reconstructs Raft volatile
state; it is a deterministic process-restart model, not torn-write simulation.
The SQL integration test performs clean file close/reopen. These additions do not
establish power-loss durability for TDiskState or SQLite, authenticated pairing
of stores, exhaustive fault schedules, snapshots or retry deduplication. Those
and independent review remain production gates.
Loading
Loading